From a71474ce288c332232d7d442a5783c0f0192a9df Mon Sep 17 00:00:00 2001 From: TerryByte Date: Thu, 3 Sep 2026 18:13:11 -0400 Subject: [PATCH] docs: reorder running log newest-first, add clear entry separators --- running_log.md | 59 ++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 26 deletions(-) diff --git a/running_log.md b/running_log.md index 4ae257e..fa504cc 100644 --- a/running_log.md +++ b/running_log.md @@ -1,34 +1,12 @@ +# Running Log -## 2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller) +Most recent entries at the top, oldest at the bottom. Each entry separated by a horizontal rule. -**Symptom:** After a full reboot of Net-Controller, `mail.infernalaquatics.com` was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy. - -**Investigation:** -- Confirmed all 18 `mailcowdockerized-*` containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host. -- Mailcow's nginx web UI (port 8087) responded `HTTP 200` locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out. -- Found mailcow's built-in network isolation firewall rule (`MAILCOW` chain in `iptables FORWARD`) drops any forwarded traffic not entering via the `br-mailcow` bridge. NPM lives on a separate docker bridge, so its proxied requests to `192.168.1.3:8087` (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it. - -**Fix (Option 1 — network attach, no firewall changes):** -- Attached the `npm` container to `mailcowdockerized_mailcow-network` (`docker network connect`). -- Updated the `mail.infernalaquatics.com` proxy host's Forward Hostname/IP in NPM from `192.168.1.3` to the mailcow nginx container name (`mailcowdockerized-nginx-mailcow-1`), keeping port `8087`, so traffic routes container-to-container instead of crossing the isolated LAN bridge. - -**Regression hit during the fix:** -- Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable"). -- Fixed by disconnecting and reconnecting NPM to the mailcow network with `--gw-priority -1`, so the original `proxy-manager_npm-net` gateway remains the default route while still allowing reachability into the mailcow network. - -**Second issue — stale generated nginx config:** -- After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's *generated* nginx config file (`/data/nginx/proxy_host/3.conf`) still containing the old LAN IP (`set $server "192.168.1.3";`), i.e. the DB change never propagated to the on-disk config via a restart alone. -- Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (`mailcowdockerized-nginx-mailcow-1`). - -**Verification:** -- `curl https://mail.infernalaquatics.com` → consistent `HTTP 200`, ~25-35ms response time across multiple attempts. -- Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up". - -**Takeaway for next time:** When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated `.conf` file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes. +--- ## 2026-09-03 — CrowdSec down after reboot + long-standing firewall-bouncer auth failure (Net-Controller) -**Symptom:** CrowdSec showing "Down" in Uptime Kuma after the earlier full host reboot (same reboot that broke mailcow reachability, see previous entry). +**Symptom:** CrowdSec showing "Down" in Uptime Kuma after the earlier full host reboot (same reboot that broke mailcow reachability, see entry below). ### Issue 1 — CrowdSec engine container not running @@ -71,3 +49,32 @@ - CrowdSec's own container needed an explicit `RestartPolicy` fix (`unless-stopped`) that most of the other stack containers already had — worth spot-checking other standalone/non-compose-managed containers on this host for the same gap. - The firewall-bouncer is a **host-level systemd service**, not a docker container — easy to overlook when doing container-focused health checks (`docker ps` alone won't show it). Add it to the regular health-check rotation going forward. - Bouncer API keys are one-time-display (`cscli bouncers add` only shows the key once) and are independent of the CrowdSec engine's own restarts — if the engine's bouncer registry is ever wiped/reset, all bouncer services depending on old keys will silently start failing until manually re-registered. Consider a monitoring check on `cscli bouncers list` validity or on the bouncer's own systemd restart count as an early-warning signal for this specific failure mode. + +--- + +## 2026-09-03 — Mailcow unreachable after full host reboot (Net-Controller) + +**Symptom:** After a full reboot of Net-Controller, `mail.infernalaquatics.com` was unreachable externally (timeouts on both HTTP and HTTPS), despite all mailcow containers reporting healthy. + +**Investigation:** +- Confirmed all 18 `mailcowdockerized-*` containers up and healthy post-reboot; mail ports (25/465/587/143/993/110/995) listening on host. +- Mailcow's nginx web UI (port 8087) responded `HTTP 200` locally on the host, but NPM (Nginx Proxy Manager) could not reach it — requests timed out. +- Found mailcow's built-in network isolation firewall rule (`MAILCOW` chain in `iptables FORWARD`) drops any forwarded traffic not entering via the `br-mailcow` bridge. NPM lives on a separate docker bridge, so its proxied requests to `192.168.1.3:8087` (the LAN IP) were being silently dropped by this isolation rule after the reboot re-applied it. + +**Fix (Option 1 — network attach, no firewall changes):** +- Attached the `npm` container to `mailcowdockerized_mailcow-network` (`docker network connect`). +- Updated the `mail.infernalaquatics.com` proxy host's Forward Hostname/IP in NPM from `192.168.1.3` to the mailcow nginx container name (`mailcowdockerized-nginx-mailcow-1`), keeping port `8087`, so traffic routes container-to-container instead of crossing the isolated LAN bridge. + +**Regression hit during the fix:** +- Attaching NPM to a second docker network flipped its default outbound route to the mailcow bridge gateway (no internet access), breaking NPM's own outbound connectivity (Let's Encrypt renewals, IP-range fetches all started failing with "Network is unreachable"). +- Fixed by disconnecting and reconnecting NPM to the mailcow network with `--gw-priority -1`, so the original `proxy-manager_npm-net` gateway remains the default route while still allowing reachability into the mailcow network. + +**Second issue — stale generated nginx config:** +- After the DB record was updated to point at the container name, the proxy host still failed intermittently — traced to NPM's *generated* nginx config file (`/data/nginx/proxy_host/3.conf`) still containing the old LAN IP (`set $server "192.168.1.3";`), i.e. the DB change never propagated to the on-disk config via a restart alone. +- Resolved by re-saving the proxy host in the NPM web UI (Details tab, no actual value change needed) — this forced NPM to regenerate the config file from the DB and reload nginx with the correct upstream (`mailcowdockerized-nginx-mailcow-1`). + +**Verification:** +- `curl https://mail.infernalaquatics.com` → consistent `HTTP 200`, ~25-35ms response time across multiple attempts. +- Confirmed via Uptime Kuma dashboard: mail.infernalaquatics.com back to "Up". + +**Takeaway for next time:** When touching NPM's forward_host in the DB directly (vs. through the UI), always re-save through the UI (or explicitly restart+verify the generated `.conf` file) to guarantee the config regenerates — a raw container restart is not sufficient to pick up DB-only changes.