lab-ops/running_log.md

Running Log

Most recent entries at the top, oldest at the bottom. Each entry separated by a horizontal rule.

---

2026-09-03 — CrowdSec down after reboot + long-standing firewall-bouncer auth failure (Net-Controller)

Symptom: CrowdSec showing "Down" in Uptime Kuma after the earlier full host reboot (same reboot that broke mailcow reachability, see entry below).

Issue 1 — CrowdSec engine container not running

Investigation:

• docker ps -a showed crowdsec container Exited (0) ~17 hours prior — a clean shutdown (SIGTERM received, shutting down), not a crash.
• docker inspect crowdsec confirmed exit code 0, no OOM kill, no error — it simply received the reboot's shutdown signal and never came back.
• Root cause: docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' showed RestartPolicy: no — unlike the rest of the stack, this container had no auto-restart policy configured, so it stayed down after the reboot instead of self-healing like every other service on the host.
• Container's compose label pointed to /data/compose/14, which no longer exists on disk (stale label from original provisioning, likely Coolify/Portainer-managed at some point) — not pursued further since fixing the running container directly was simpler and safer.

Fix:

• docker start crowdsec
• docker update --restart unless-stopped crowdsec — ensures it now survives future reboots automatically.
• Verified: container came up clean, re-downloaded/updated its hub (parsers, scenarios, collections — flagged one outdated collection crowdsecurity/http-cve, auto-updated to CVE-2022-44877 v0.4), started processing normally, confirmed pulling from CrowdSec Central API.

Issue 2 — firewall-bouncer crash-looping since ~Aug 18 (pre-existing, unrelated to reboot)

Discovery: While verifying CrowdSec's engine logs, noticed the crowdsec-firewall-bouncer client was receiving 403 Forbidden on every single /v1/decisions/stream poll (every 10s), both before and after the reboot — meaning this was a standing issue, not caused by tonight's outage.

Investigation:

• The bouncer is not a docker container — it's a host-level systemd service: crowdsec-firewall-bouncer.service (installed via crowdsec-firewall-bouncer-iptables package, config at /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml).
• systemctl status showed it activating (auto-restart) — crash-looping continuously.
• journalctl -u crowdsec-firewall-bouncer.service showed the same fatal error on every cycle: level=fatal msg="process terminated with error: bouncer stream halted", immediately followed by systemd restarting it (10s backoff).
• Scale of the problem: restart counter was at 5,829 by the time this was caught, with log rotation files dating back to Aug 18, 2026 — meaning this bouncer had been non-functional (crashing every ~10s) for roughly 2.5 weeks before discovery.
• Root cause confirmed via docker exec crowdsec cscli bouncers list on the engine side — zero bouncers registered. The API key configured in the bouncer's yaml (api_key: MRPnN0XqxrVOspREia3c4jvRYfi+/Cr/9bUKAcxdoWg) did not correspond to any bouncer known to the CrowdSec engine (likely invalidated by a prior CrowdSec engine reset/reinstall, or the bouncer was never properly registered after an update).
• Security impact: with the bouncer unable to authenticate, ban decisions from CrowdSec's engine (local detections + CrowdSec Central API community blocklist) were never being enforced at the firewall (nftables) layer for the ~2.5 week window. CrowdSec's detection/logging continued working the whole time (it runs independently), but actual IP blocking was not happening.

Fix:

1. Generated a fresh bouncer API key on the engine: docker exec crowdsec cscli bouncers add firewall-bouncer-netcontroller → new key DkIEnS1TfGfly49gAAb4cbn1T6/+EZBVSEFCoEojt0M.
2. Backed up the existing config: /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.bak.<timestamp>.
3. Replaced the api_key: line in /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml with the new key via sed.
4. sudo systemctl restart crowdsec-firewall-bouncer.service.

Verification:

• systemctl is-active → active and stable (previously dying in under a second on every cycle; now running continuously with no restarts).
• cscli bouncers list on the engine now shows firewall-bouncer-netcontroller with Valid: ✔️, IP 172.18.0.1, auth type api-key, and a fresh Last API pull timestamp.
• CrowdSec engine logs confirmed the bouncer's /v1/decisions/stream polls now return HTTP 200 (was 403) every 10s as expected.
• Confirmed actual enforcement is live: sudo nft list ruleset shows the ip crowdsec and ip6 crowdsec6 tables (matching the nftables: config block — note: NOT under the inet family, took one wrong guess before finding the right table family) with active, incrementing packet/byte counters on the block chains (17,990+ packets processed at time of check).

Takeaways for next time:

• CrowdSec's own container needed an explicit RestartPolicy fix (unless-stopped) that most of the other stack containers already had — worth spot-checking other standalone/non-compose-managed containers on this host for the same gap.
• The firewall-bouncer is a host-level systemd service, not a docker container — easy to overlook when doing container-focused health checks (docker ps alone won't show it). Add it to the regular health-check rotation going forward.
• Bouncer API keys are one-time-display (cscli bouncers add only shows the key once) and are independent of the CrowdSec engine's own restarts — if the engine's bouncer registry is ever wiped/reset, all bouncer services depending on old keys will silently start failing until manually re-registered. Consider a monitoring check on cscli bouncers list validity or on the bouncer's own systemd restart count as an early-warning signal for this specific failure mode.

---

2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller)

Symptom: After a full reboot of Net-Controller, mail.infernalaquatics.com was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy.

Investigation:

• Confirmed all 18 mailcowdockerized-* containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host.
• Mailcow's nginx web UI (port 8087) responded HTTP 200 locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out.
• Found mailcow's built-in network isolation firewall rule (MAILCOW chain in iptables FORWARD) drops any forwarded traffic not entering via the br-mailcow bridge. NPM lives on a separate docker bridge, so its proxied requests to 192.168.1.3:8087 (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it.

Fix (Option 1 — network attach, no firewall changes):

• Attached the npm container to mailcowdockerized_mailcow-network (docker network connect).
• Updated the mail.infernalaquatics.com proxy host's Forward Hostname/IP in NPM from 192.168.1.3 to the mailcow nginx container name (mailcowdockerized-nginx-mailcow-1), keeping port 8087, so traffic routes container-to-container instead of crossing the isolated LAN bridge.

Regression hit during the fix:

• Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable").
• Fixed by disconnecting and reconnecting NPM to the mailcow network with --gw-priority -1, so the original proxy-manager_npm-net gateway remains the default route while still allowing reachability into the mailcow network.

Second issue — stale generated nginx config:

• After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's generated nginx config file (/data/nginx/proxy_host/3.conf) still containing the old LAN IP (set $server "192.168.1.3";), i.e. the DB change never propagated to the on-disk config via a restart alone.
• Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (mailcowdockerized-nginx-mailcow-1).

Verification:

• curl https://mail.infernalaquatics.com → consistent HTTP 200, ~25-35ms response time across multiple attempts.
• Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up".

Takeaway for next time: When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated .conf file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes.