Incident: hyper.media outage on 2026-09-08
A Watchtower auto-deploy of release 2026.9.4 stopped the daemon container and failed to recreate it because host port 56000 was still bound. The container was left in Created state and Watchtower stopped tracking it, so the site stayed down for 13 minutes until it was started manually.

Incident: hyper.media outage on 2026-09-08

Summary

On 2026-09-08, hyper.media served no traffic from 08:24:27 to 08:37:33 UTC — about 13 minutes. The cause was an automated deployment of release 2026.9.4 that half-completed: Watchtower stopped the daemon_gateway container, then failed to create its replacement because host port 56000 was still bound by the container it had just stopped. Docker refused the port binding, the new container was left in Created state, and it never started. web_gateway came up on the new image but had no daemon to query, so every route hung.

The outage did not resolve itself and would not have. Watchtower only scans running containers, so the moment the daemon was stranded in Created it fell out of the deploy tool's view — its scan count dropped from 5 containers to 4 and stayed there through two further cycles. The container's restart: unless-stopped policy did not apply either, because restart policies act on containers that exit, and this one never started. Service was restored by manually running docker start daemon_gateway.

This is the second production incident caused by a silent Watchtower failure, after the July incident in which Watchtower crashed and stopped deploying for three days.

Timeline (UTC)

    08:18:10 — Watchtower session completes normally: Failed=0 Scanned=5 Updated=2.

    08:23:45 – 08:24:27 — Watchtower detects three new images for release 2026.9.4: web:latest (90b2d79e8ba3), notify:latest (05fd3367bdf2), site:latest (44e7e1f8f3c0).

    08:24:27 — Watchtower stops daemon_gateway (d56511fe39aa) with SIGTERM. The outage begins here — this is the process that serves all document data.

    08:24:58 — notify stopped. 08:25:28 — web_gateway stopped.

    08:25:58 — web_gateway and notify recreated successfully.

    08:25:59 — Watchtower creates daemon_gateway and Docker rejects it: failed to set up container networking: driver failed programming external connectivity on endpoint daemon_gateway: failed to bind host port for 0.0.0.0:56000:172.18.0.5:56000/tcp: address already in use. The container is left in Created state with ExitCode=128.

    08:26:08 — Session done Failed=1 Scanned=5 Updated=2. Watchtower records the failure and moves on.

    08:28:07 — Session done Failed=0 Scanned=4 Updated=0. The scan count has dropped to 4: the stranded container is no longer visible to Watchtower.

    08:33:07 — Session done Failed=0 Scanned=4 Updated=0. Still invisible, still no retry.

    ~08:36 — Outage noticed by a person loading the site. Investigation begins. External requests to https://hyper.media/ return no response within 20 seconds.

    08:37:22 — docker start daemon_gateway run manually. Ports 56000 and 56001 were free by this point.

    08:37:33 — DaemonStarted on [::]:56000 and [::]:56002. Site begins serving. Outage ends.

    08:38:07 — Session done Scanned=5. The container is back in Watchtower's scope, confirming the mechanism that had hidden it.

Observed symptoms

    External requests to https://hyper.media/ and https://hyper.media/hm/api/config returned no HTTP response, timing out at 20 seconds.

    docker ps looked deceptively healthy: web_gateway, notify, relay and vault were all Up. Only docker ps -a revealed daemon_gateway with status Created.

    Host load average was 0.73 — unusually low, because the process that normally does the work was not running. Resource dashboards would have shown a calm, healthy machine.

    daemon_gateway logs were completely empty, because the container had never started.

Root cause analysis

1. Port-release race between stop and recreate (primary)

daemon_gateway is the only container in this stack that publishes host ports: 56000/tcp, and 56001 on both TCP and UDP. Watchtower's update sequence stops the old container and then creates the new one. Here those steps were 92 seconds apart (08:24:27 → 08:25:59), and the host port was still held when the new container was created — the userspace docker-proxy process for the old container had not released it.

Docker treats a port-binding failure at creation time as fatal for that container. It does not retry, and the container is parked in Created — a state in which it has never executed a single instruction.

The other three services in the deploy were unaffected because they publish no host ports and therefore have no contended resource.

2. Why it could not self-heal (why the outage was open-ended)

Three independent recovery mechanisms that might plausibly have saved this all failed to apply:

    Watchtower stopped tracking the container. Watchtower enumerates running containers. Once the daemon was stranded in Created, it left the scan set — visible in the session logs as Scanned=5 before the failure, Scanned=4 during, and Scanned=5 again immediately after the manual start. It was not waiting to retry; it no longer knew the container existed.

    The restart policy did not fire. The container is configured restart: unless-stopped, which sounds like it should cover this. But RestartCount=0 and StartedAt=0001-01-01T00:00:00Z show it never entered the running state, and Docker restart policies act on the exit of a running container. A never-started container is out of scope. (A Docker daemon or host restart would have started it, since unless-stopped containers are started on daemon boot — but nothing was scheduled to do that.)

    No retry job exists. The host's only scheduled tasks are e2scrub_all and sysstat. Nothing periodically reconciles container state against the compose file.

3. Detection gap (contributing to duration)

Nothing alerted. The outage was discovered by a person loading the site roughly 12 minutes in. Of the 13-minute outage, about 11 minutes was time-to-detection and under 2 minutes was time-to-fix. The failure mode is also actively self-concealing: a container in Created does not appear in docker ps, host load drops rather than spikes, and the deploy tool removes the container from its own view — so the three places an operator or a system would normally look all read as healthy.

4. Ruled out

    Resource exhaustion — load average 0.73, no memory or disk pressure. The machine was idle because the daemon was absent.

    A crash loop or bad image — RestartCount=0; the container never ran. The same 2026.9.4 image started cleanly and has run without incident since.

    A port conflict with traefik — traefik does listen on *:56002, but daemon_gateway publishes only 56000 and 56001, so there is no standing conflict. Ports 56000 and 56001 were confirmed free before the manual start.

    A migration or data problem — the daemon logged MIGRATING briefly at boot, but no reindex ran and it reached normal operation immediately, with zero panic or fatal log lines.

Recovery

docker start daemon_gateway completed the interrupted deployment. No configuration or image change was required — the failure was purely that a container Watchtower intended to start had not been started.

Post-recovery measurements confirmed a healthy system: homepage latency fell from 5.4 s to 1.4 s to 0.78 s as caches warmed, /hm/api/config returned in 0.36 s, load settled at 1.80, and CPU sat at 31% (daemon) and 25% (web). The deployed release was verified as 2026.9.4, commit a1dbf9018d362977e1260b2acfb48b8656e768e8, with web and daemon in sync.

Recommended fixes

In order of how much risk each removes:

    Stop publishing the daemon's ports on the host. Route 56000 through the internal Docker network like every other service in the stack, with traefik fronting anything that must be externally reachable. This eliminates the contended resource, and with it this entire class of failure. It is the only change that makes the race impossible rather than merely less likely.

    Add external uptime monitoring for hyper.media. The cheapest fix by a wide margin, and it addresses the 11 minutes of the outage that were detection rather than repair. Any external check would have fired within a minute.

    Reconcile container state after deploys. Watchtower will not retry a failed create and cannot see the container it stranded. A periodic check for containers in non-running states — or a post-deploy verification step that compares the compose file against running containers — closes the gap without replacing the deploy tool.

    Reconsider unattended auto-deploy for the daemon. Watchtower has now caused two production incidents through silent failure: this one, and the July incident where it crashed and stopped deploying for three days. Both were invisible until a human noticed. The pattern is not a specific bug so much as a tool that fails quietly in a position where quiet failure is expensive.

Evidence

    Watchtower session logs showing the failed create and the Scanned=5 → 4 → 5 transition around it.

    docker inspect daemon_gateway: Status=created, ExitCode=128, RestartCount=0, StartedAt=0001-01-01T00:00:00Z, RestartPolicy=unless-stopped, and the full port-binding error string.

    ss -tln confirming ports 56000 and 56001 were unbound before the manual start, and that traefik's 56002 listener does not overlap the daemon's published ports.

    Daemon startup logs from 08:37:22–08:37:33 showing StartupWALDrained, P2PNodeStarted, and DaemonStarted, with no panics.

    Host crontab and systemd timers showing no docker-related retry mechanism.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

People
No additional collaborators