lab-ops/running_log.md

2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller)

Symptom: After a full reboot of Net-Controller, mail.infernalaquatics.com was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy.

Investigation:

• Confirmed all 18 mailcowdockerized-* containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host.
• Mailcow's nginx web UI (port 8087) responded HTTP 200 locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out.
• Found mailcow's built-in network isolation firewall rule (MAILCOW chain in iptables FORWARD) drops any forwarded traffic not entering via the br-mailcow bridge. NPM lives on a separate docker bridge, so its proxied requests to 192.168.1.3:8087 (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it.

Fix (Option 1 — network attach, no firewall changes):

• Attached the npm container to mailcowdockerized_mailcow-network (docker network connect).
• Updated the mail.infernalaquatics.com proxy host's Forward Hostname/IP in NPM from 192.168.1.3 to the mailcow nginx container name (mailcowdockerized-nginx-mailcow-1), keeping port 8087, so traffic routes container-to-container instead of crossing the isolated LAN bridge.

Regression hit during the fix:

• Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable").
• Fixed by disconnecting and reconnecting NPM to the mailcow network with --gw-priority -1, so the original proxy-manager_npm-net gateway remains the default route while still allowing reachability into the mailcow network.

Second issue — stale generated nginx config:

• After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's generated nginx config file (/data/nginx/proxy_host/3.conf) still containing the old LAN IP (set $server "192.168.1.3";), i.e. the DB change never propagated to the on-disk config via a restart alone.
• Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (mailcowdockerized-nginx-mailcow-1).

Verification:

• curl https://mail.infernalaquatics.com → consistent HTTP 200, ~25-35ms response time across multiple attempts.
• Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up".

Takeaway for next time: When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated .conf file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes.