docs: log mailcow reachability incident and fix (2026-09-03)

2026-09-03 17:50:15 -04:00
parent edce13587a
commit f31ac54675
2 changed files with 27 additions and 0 deletions
@@ -0,0 +1,27 @@
+
+## 2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller)
+
+**Symptom:** After a full reboot of Net-Controller, `mail.infernalaquatics.com` was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy.
+
+**Investigation:**
+- Confirmed all 18 `mailcowdockerized-*` containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host.
+- Mailcow's nginx web UI (port 8087) responded `HTTP 200` locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out.
+- Found mailcow's built-in network isolation firewall rule (`MAILCOW` chain in `iptables FORWARD`) drops any forwarded traffic not entering via the `br-mailcow` bridge. NPM lives on a separate docker bridge, so its proxied requests to `192.168.1.3:8087` (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it.
+
+**Fix (Option 1 — network attach, no firewall changes):**
+- Attached the `npm` container to `mailcowdockerized_mailcow-network` (`docker network connect`).
+- Updated the `mail.infernalaquatics.com` proxy host's Forward Hostname/IP in NPM from `192.168.1.3` to the mailcow nginx container name (`mailcowdockerized-nginx-mailcow-1`), keeping port `8087`, so traffic routes container-to-container instead of crossing the isolated LAN bridge.
+
+**Regression hit during the fix:**
+- Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable").
+- Fixed by disconnecting and reconnecting NPM to the mailcow network with `--gw-priority -1`, so the original `proxy-manager_npm-net` gateway remains the default route while still allowing reachability into the mailcow network.
+
+**Second issue — stale generated nginx config:**
+- After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's *generated* nginx config file (`/data/nginx/proxy_host/3.conf`) still containing the old LAN IP (`set $server "192.168.1.3";`), i.e. the DB change never propagated to the on-disk config via a restart alone.
+- Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (`mailcowdockerized-nginx-mailcow-1`).
+
+**Verification:**
+- `curl https://mail.infernalaquatics.com` → consistent `HTTP 200`, ~25-35ms response time across multiple attempts.
+- Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up".
+
+**Takeaway for next time:** When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated `.conf` file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes.