Incident: hyper.media outage on 2026-07-09
Production web gateway wedged after a burst of failed daemon RPCs at 13:57 UTC and was recovered by manual container restarts at 14:17-14:19. Chronic daemon CPU saturation from scanner-triggered P2P discovery set the stage; an ACME retry storm for stale custom domains is a separate ongoing issue.

Incident: hyper.media outage on 2026-07-09

Summary

On 2026-07-09, hyper.media stopped responding to external requests from roughly 14:05 to 14:20 UTC. The acute failure began with a burst of failed daemon RPCs at 13:57–13:59 (log volume spiked 6x, with ~1,650 ConnectError stack dumps in under 4 minutes), after which web_gateway throughput decayed steadily — from ~3,500 completed log lines/minute down to ~1,000 — until it was manually restarted by team members at 14:17:33 (the daemon followed at 14:18:46). The site did not recover on its own, and the outage was not a simple request-rate problem: scanner traffic at the time (~33 junk requests/minute) was lower than rates the site handles fine in healthy periods. The underlying condition — the daemon chronically pegged at ~320–360% CPU on a 4-core box, largely doing recursive P2P discovery for scanner junk paths, with discovery latencies of 5–60 seconds — persists after recovery and makes the web app fragile to bursts. A separate ongoing issue: traefik endlessly retries Let's Encrypt certificates for stale custom domains whose DNS points elsewhere, burning ACME rate-limit budget.

Timeline (UTC)


    ~10:40 — daemon_gateway container was recreated (uptime 4h at time of diagnosis; all other containers had 41h uptime; watchtower shows no image updates in 24h, image dated 2026-06-22, so this was a manual or external restart).

    13:57–13:59 — web_gateway log volume spiked from ~3,500 to 24,165 lines/minute, dominated by ~1,650 multi-line ConnectError stack dumps from failed gRPC calls to the daemon. This is the acute trigger.

    14:00–14:17 — web_gateway throughput decayed steadily (2,690 → 1,007 completed log lines/minute); externally the site appeared down. Monitoring flagged the outage at ~14:05. At 14:14 an external check of https://hyper.media/ timed out after 30 seconds with no HTTP response.

    14:09 and 14:16 — Two team members logged into the server via SSH.

    14:17:33 — web_gateway container manually restarted; 14:18:46 — daemon_gateway manually restarted. Service recovered immediately after (~14:20). An earlier revision of this report incorrectly stated the service "recovered on its own" — kernel logs and container start times show the manual restarts.

Observed symptoms


    External requests to https://hyper.media timed out (no response within 30s).

    Legitimate API calls were extremely slow: a ListCapabilities request took 26 seconds.

    Server-side rendering loaders failed with Error: Timed out from try-until-success.ts via discovery.ts, for both real documents and junk paths.

    web_gateway logged ~26,666 lines in 10 minutes, including ~2,700 GET /hm/api/config requests in 10 minutes, many with no recorded response time (hung or aborted).

Root cause analysis

1. Acute trigger: web_gateway wedged after an RPC failure burst (primary)

At 13:57–13:59 UTC the web app suffered a cascade of ~1,650 failed gRPC (ConnectError) calls to the daemon in under four minutes — 94% of them not_found (1,558), only 64 timeouts. The request rate during the burst was only modestly elevated (~1,700 GETs over the window), so this was a burst of lookups for documents that don't exist — consistent with a scanner sweep across junk paths and/or rendering pages dense with dead hm:// links, each miss both dumping a stack trace and queuing recursive discovery work in the already-saturated daemon. From then until the 14:17 restart, web_gateway completed steadily less work each minute despite normal incoming traffic — the signature of a Node.js process wedged on accumulated in-flight state (pending upstream calls, held sockets, retry loops in try-until-success) rather than one that is merely busy. A restart of the container cleared it instantly. The exact internal mechanism (upstream connection-pool exhaustion, event-loop starvation, or unbounded retry accumulation) still needs a code-level investigation.

2. Chronic condition: scanner traffic amplified into P2P discovery work (contributing)

Internet vulnerability scanners were probing hosted sites with junk paths: phpinfo.php, wp-admin/*, *.env (in web/, backend/, frontend/, server/, src/, core/, admin/, public/ variants), moadmin.php, users.php, and similar. There were 1,957 scanner-path hits in one hour on the web gateway.

Each of these requests misses the local database, and the web app's document loader then triggers a recursive P2P network discovery in the daemon (discover start: iri=hm://.../wp-admin/phpinfo.php recursive=true). Each discovery attempt ran 5–25 seconds before failing with outcome=error, and they queued continuously.

Result: daemon_gateway consumes ~320–360% CPU on a 4-core machine (load average ~5) — and importantly, it does so chronically, including during healthy operation. Measurements after recovery showed a higher scanner rate (~43 junk requests/minute vs ~33 during the outage hour) and slower discovery (up to 58s) while the site served fine. So this load did not by itself flip the site from up to down; it is the standing condition that makes every daemon RPC slow and leaves the web app with no headroom to absorb a burst.

Notably, this is not a flood of real user traffic: legitimate request volume is modest. The amplification factor (one cheap scanner GET → many seconds of daemon CPU and P2P dialing) is what turns background internet noise into sustained saturation.

3. ACME certificate retry storm (separate ongoing issue)

Traefik was continuously retrying Let's Encrypt certificate issuance for custom domains whose DNS no longer points at this server: botical.com, bad.botical.com, joe2.botical.com, nataliamarin.blog, adrianamerlo.com, alicantebitcoinplebs.com. Challenges for these domains reached a foreign IP (23.20.24.146, AWS) and failed with tls: unrecognized name.

Let's Encrypt responded with 429 rate-limit errors ("too many failed authorizations (5) in the last 1h"). Because issuance can never succeed for these domains, traefik retries indefinitely, adding constant edge churn and risking rate-limit exhaustion for the account, which could eventually block certificate renewal for legitimate domains.

4. Ruled out


    Memory: 23 GiB of 30 GiB available, no swap in use, no OOM kills (OOMKilled=false).

    Disk: root filesystem at 45% (53 GB free).

    Container crashes: no crash before the outage; the mid-outage restarts at 14:17–14:19 were manual remediation.

    Real user surge: top requested paths were routine API/config endpoints plus scanner noise.

    Scanner request rate as the acute cause: ~33 junk requests/minute during the outage hour vs ~43/minute measured post-recovery with the site healthy — the rate alone does not explain the outage.

    Kernel conntrack exhaustion: 2,996 of 262,144 entries in use — nowhere near the limit.

Evidence snapshots


    docker stats: daemon_gateway 361% CPU / 3.7 GiB RAM; web_gateway 24% CPU / 2.3 GiB RAM.

    Daemon logs: continuous seed/synctrace discovery cycles for scanner paths, each 5–25s, outcome=error, alongside BootstrapFinished rounds with 8/10 peer-dial failures (context deadline exceeded).

    Traefik logs: repeating Unable to obtain ACME certificate errors with 429 rateLimited for the stale domains listed above.

    TCP state: 654 established connections, 315 in time-wait — elevated but not exhausted.

Prevention and mitigation

Fix the amplification (root fix)


    Don't trigger P2P discovery for implausible paths. The web loader should never launch discovery for requests matching known scanner signatures: *.php, *.asp, .env (and */.env), wp-admin/*, wp-content/*, .git/*, cgi-bin/*, etc. Return a fast 404 instead. This single change removes almost all of the amplification.

    Cache negative discovery results. When discovery fails for an IRI, remember that failure (even 5–15 minutes) so repeated probes of the same junk path cost one lookup, not one 25-second P2P search per hit.

    Cap concurrent discoveries in the daemon and put them in a bounded queue with a global budget, so discovery load can never starve the CPU needed to serve documents that already exist locally.

    Make SSR degrade instead of hang. Loaders should serve a fast "not found / still syncing" response rather than blocking the request thread on try-until-success while discovery runs.

Harden the edge (stopgaps, deployable today)


    Block scanner paths at traefik with a router rule or middleware that returns 404/403 for the signature paths above, before they ever reach the web app.

    Rate-limit per client IP at traefik so a single scanner cannot issue hundreds of expensive requests per minute.

    Alert on daemon CPU (e.g. Prometheus alert at sustained >300% for 5 minutes) so the team sees saturation before external monitoring reports a full outage.

Stop the ACME retry storm


    Remove custom-domain routes whose DNS no longer points at the server (botical.com, bad.botical.com, joe2.botical.com, nataliamarin.blog, adrianamerlo.com, alicantebitcoinplebs.com). Issuance for them can never succeed and each retry burns Let's Encrypt rate-limit budget shared with legitimate domains.

    Automate the cleanup: periodically verify that each configured custom domain still resolves to this server, and disable ACME for those that don't, instead of retrying forever.

Operational follow-ups


    Investigate why daemon_gateway was recreated at 10:40 UTC — no image update occurred, so someone or something restarted it manually.

    Consider more CPU headroom: 4 cores is thin when a single misbehaving workload can consume 3.6 of them.

    Add an external synthetic check that alerts on p95 latency, not just hard downtime — this incident was a slow-burn saturation that flapped in and out of "down".

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

People
No additional collaborators