## 2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller)

**Symptom:** After a full reboot of Net-Controller, `mail.infernalaquatics.com` was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy.

**Investigation:**
- Confirmed all 18 `mailcowdockerized-*` containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host.
- Mailcow's nginx web UI (port 8087) responded `HTTP 200` locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out.
- Found mailcow's built-in network isolation firewall rule (`MAILCOW` chain in `iptables FORWARD`) drops any forwarded traffic not entering via the `br-mailcow` bridge. NPM lives on a separate docker bridge, so its proxied requests to `192.168.1.3:8087` (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it.

**Fix (Option 1 — network attach, no firewall changes):**
- Attached the `npm` container to `mailcowdockerized_mailcow-network` (`docker network connect`).
- Updated the `mail.infernalaquatics.com` proxy host's Forward Hostname/IP in NPM from `192.168.1.3` to the mailcow nginx container name (`mailcowdockerized-nginx-mailcow-1`), keeping port `8087`, so traffic routes container-to-container instead of crossing the isolated LAN bridge.

**Regression hit during the fix:**
- Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable").
- Fixed by disconnecting and reconnecting NPM to the mailcow network with `--gw-priority -1`, so the original `proxy-manager_npm-net` gateway remains the default route while still allowing reachability into the mailcow network.

**Second issue — stale generated nginx config:**
- After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's *generated* nginx config file (`/data/nginx/proxy_host/3.conf`) still containing the old LAN IP (`set $server "192.168.1.3";`), i.e. the DB change never propagated to the on-disk config via a restart alone.
- Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (`mailcowdockerized-nginx-mailcow-1`).

**Verification:**
- `curl https://mail.infernalaquatics.com` → consistent `HTTP 200`, ~25-35ms response time across multiple attempts.
- Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up".

**Takeaway for next time:** When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated `.conf` file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes.

## 2026-09-03 — CrowdSec down after reboot + long-standing firewall-bouncer auth failure (Net-Controller)

**Symptom:** CrowdSec showing "Down" in Uptime Kuma after the earlier full host reboot (same reboot that broke mailcow reachability, see previous entry).

### Issue 1 — CrowdSec engine container not running

**Investigation:**
- `docker ps -a` showed `crowdsec` container `Exited (0)` ~17 hours prior — a clean shutdown (`SIGTERM received, shutting down`), not a crash.
- `docker inspect crowdsec` confirmed exit code `0`, no OOM kill, no error — it simply received the reboot's shutdown signal and never came back.
- Root cause: `docker inspect --format '{{.HostConfig.RestartPolicy.Name}}'` showed `RestartPolicy: no` — unlike the rest of the stack, this container had no auto-restart policy configured, so it stayed down after the reboot instead of self-healing like every other service on the host.
- Container's compose label pointed to `/data/compose/14`, which no longer exists on disk (stale label from original provisioning, likely Coolify/Portainer-managed at some point) — not pursued further since fixing the running container directly was simpler and safer.

**Fix:**
- `docker start crowdsec`
- `docker update --restart unless-stopped crowdsec` — ensures it now survives future reboots automatically.
- Verified: container came up clean, re-downloaded/updated its hub (parsers, scenarios, collections — flagged one outdated collection `crowdsecurity/http-cve`, auto-updated to CVE-2022-44877 v0.4), started processing normally, confirmed pulling from CrowdSec Central API.

### Issue 2 — firewall-bouncer crash-looping since ~Aug 18 (pre-existing, unrelated to reboot)

**Discovery:** While verifying CrowdSec's engine logs, noticed the `crowdsec-firewall-bouncer` client was receiving `403 Forbidden` on every single `/v1/decisions/stream` poll (every 10s), both before and after the reboot — meaning this was a standing issue, not caused by tonight's outage.

**Investigation:**
- The bouncer is **not** a docker container — it's a host-level systemd service: `crowdsec-firewall-bouncer.service` (installed via `crowdsec-firewall-bouncer-iptables` package, config at `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml`).
- `systemctl status` showed it `activating (auto-restart)` — crash-looping continuously.
- `journalctl -u crowdsec-firewall-bouncer.service` showed the same fatal error on every cycle: `level=fatal msg="process terminated with error: bouncer stream halted"`, immediately followed by systemd restarting it (10s backoff).
- **Scale of the problem:** restart counter was at **5,829** by the time this was caught, with log rotation files dating back to **Aug 18, 2026** — meaning this bouncer had been non-functional (crashing every ~10s) for roughly **2.5 weeks** before discovery.
- Root cause confirmed via `docker exec crowdsec cscli bouncers list` on the engine side — **zero bouncers registered**. The API key configured in the bouncer's yaml (`api_key: MRPnN0XqxrVOspREia3c4jvRYfi+/Cr/9bUKAcxdoWg`) did not correspond to any bouncer known to the CrowdSec engine (likely invalidated by a prior CrowdSec engine reset/reinstall, or the bouncer was never properly registered after an update).
- **Security impact:** with the bouncer unable to authenticate, ban decisions from CrowdSec's engine (local detections + CrowdSec Central API community blocklist) were **never being enforced at the firewall (nftables) layer** for the ~2.5 week window. CrowdSec's detection/logging continued working the whole time (it runs independently), but actual IP blocking was not happening.

**Fix:**
1. Generated a fresh bouncer API key on the engine: `docker exec crowdsec cscli bouncers add firewall-bouncer-netcontroller` → new key `DkIEnS1TfGfly49gAAb4cbn1T6/+EZBVSEFCoEojt0M`.
2. Backed up the existing config: `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.bak.<timestamp>`.
3. Replaced the `api_key:` line in `/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml` with the new key via `sed`.
4. `sudo systemctl restart crowdsec-firewall-bouncer.service`.

**Verification:**
- `systemctl is-active` → `active` and stable (previously dying in under a second on every cycle; now running continuously with no restarts).
- `cscli bouncers list` on the engine now shows `firewall-bouncer-netcontroller` with `Valid: ✔️`, IP `172.18.0.1`, auth type `api-key`, and a fresh `Last API pull` timestamp.
- CrowdSec engine logs confirmed the bouncer's `/v1/decisions/stream` polls now return `HTTP 200` (was `403`) every 10s as expected.
- Confirmed actual enforcement is live: `sudo nft list ruleset` shows the `ip crowdsec` and `ip6 crowdsec6` tables (matching the `nftables:` config block — note: NOT under the `inet` family, took one wrong guess before finding the right table family) with active, incrementing packet/byte counters on the block chains (17,990+ packets processed at time of check).

**Takeaways for next time:**
- CrowdSec's own container needed an explicit `RestartPolicy` fix (`unless-stopped`) that most of the other stack containers already had — worth spot-checking other standalone/non-compose-managed containers on this host for the same gap.
- The firewall-bouncer is a **host-level systemd service**, not a docker container — easy to overlook when doing container-focused health checks (`docker ps` alone won't show it). Add it to the regular health-check rotation going forward.
- Bouncer API keys are one-time-display (`cscli bouncers add` only shows the key once) and are independent of the CrowdSec engine's own restarts — if the engine's bouncer registry is ever wiped/reset, all bouncer services depending on old keys will silently start failing until manually re-registered. Consider a monitoring check on `cscli bouncers list` validity or on the bouncer's own systemd restart count as an early-warning signal for this specific failure mode.