2026-11-10: agents-stable unresponsive (transcript privacy scans on a 50 GB database)
From about 05:46 UTC on 2026-11-10 the production agents server accepted connections but answered no HTTP requests. A read-path privacy check scanned whole session transcripts from a 46-50 GB SQLite file on a 4 GB container, saturating the disk and blocking the single Bun thread. Restarting did not help, and nothing alerted for over six hours.

Incident: agents-stable unresponsive on 2026-11-10

Summary

Starting around 05:46 UTC on 2026-11-10, agents-stable on the agentic host stopped answering HTTP. It accepted TCP connections but /api/health and /api/perf timed out at 30-60 s, even from inside the container. Everything served by it went dark for interactive users: https://agent.hyper.media and the /hm/api/agents/* routes on Seed sites such as seedteamtalks.hyper.media.

The cause is BrowserPrivacy.forViewer in agents/src/browser-privacy.ts, added with the integrated web browser in PR #1138. On every API response and every collaborator or public WebSocket frame, for any session or run the viewer does not own, it reads the entire transcript to decide whether it contains private browser or memory data. The result is cached only in process memory. The database has grown to 46-50 GB against a 4 GB container on a 7.6 GB host, so each scan is a run of random disk reads. The disk sat at 98% utilisation, and because bun:sqlite is synchronous on the single Bun thread, the event loop spent nearly all its time waiting on disk and never got to accept() or write responses.

Restarting does not fix it: the privacy cache is empty after a restart, so the same scans start immediately. Nothing alerted, although the Docker healthcheck failed continuously from about 05:50.

Facts below come from the agents team's investigation on the host between 12:00 and 12:20 UTC unless marked otherwise.

Timeline (UTC)

    2026-11-03 - Database measured at 36 GB. A related read-path saturation (ListSessions busy subquery) is fixed with the runs_by_agent_status index.

    2026-11-07 to 11-09 - Separate incident: sandboxes abort on msb sandbox id 65536 and above (fix PR #1333). Alerting work opened as seed#1334 and infra#24, not yet merged.

    05:43 - Image seedhypermedia/agents:latest built from commit 30e609e0.

    05:46 - Watchtower recreates agents-stable on the new image. Outage begins about here. Boot takes about 100 s on this database.

    ~05:50 - Docker healthcheck starts failing and keeps failing for the rest of the morning.

    05:00-06:00 hour - 138 slow statements logged. 655 in the 06:00 hour, then about 500 per hour all morning. The most frequent is the transcript scan, at 300-700 ms each.

    05:50-12:00 - No alert fires. Agent runs and triggers continue slowly in the background, for example a Clerk trigger (cc4e19e9…, account z6MkrbYs…) and ion recursive-intelligence sessions. agents-dev and agents-staging on the same host stay healthy.

    ~12:00 - Investigation starts on the host.

    12:00-12:09 - Diagnosis: Caddy holds 774 established and 73 half-closed connections to :3050 from about 50 client IPs, one with 38 WebSockets. sdb at 98% utilisation, about 9,000 preads/s across the whole file. strace shows the event loop waking about twice per 3 s. Sampled disk pages are overflow pages of large tool results and durable session files.

    During diagnosis - A read-only bun -e query run through docker exec outlives its SSH timeout and adds disk load for about 3 minutes until it is killed by hand.

    12:09 - agents-stable restarted.

    12:10:45 - Server listening again.

    12:11 - Wedged again, as the cold privacy cache drives the same scans.

    12:20 onward - Three fix workstreams start in parallel (below). Outage ongoing at time of writing.

Impact

    Interactive agents were unusable on production for 6+ hours and counting: the agents API, the desktop and web agents UI, and /hm/api/agents/* on every Seed site that proxies to agents-stable.

    Health and perf endpoints timed out, so even diagnosis had to happen on the host.

    Background work degraded but continued. Triggered sessions and runs still executed, slowly, so some automations kept producing output while users could not see or steer them.

    Staging and dev were not affected. Their databases are much smaller.

    No evidence of data loss. The privacy check is a read path.

Root cause

BrowserPrivacy.forViewer sits on the read path of every API response and every collaborator or public WebSocket frame. For any session or run the viewer does not own, it decides whether the transcript contains private browser or memory data by reading all of it:

SELECT seq, event_cbor FROM session_events WHERE session_id = ? AND seq > ? ORDER BY seq

Runs do the same against run_journal. The answer is memoised in process memory only, never persisted.

This was cheap when the database fit in the page cache. At 46-50 GB on a 4 GB container, it is not:

    The transcripts are big. Sampled pages were overflow pages of large tool results (fetched web pages, arXiv XML, research markdown) and durable session files.

    The reads are random. About 9,000 preads/s spread over the whole file, with sdb at 98% utilisation.

    The reads block everything. bun:sqlite calls are synchronous on the single Bun thread. The thread sat in D (disk wait) state, so the event loop could not accept connections or send responses.

    Load multiplies the cost. About 50 clients held 774 connections, and every WebSocket frame for a shared or public session triggered the check.

    A restart resets the cache. With nothing persisted, a fresh process rescans everything on first view, so it re-wedges within a minute.

Contributing factors

    Database growth with no retention. 36 GB on 11-03, about 2 GB per day since, mostly large tool results that nothing prunes.

    Slow boot. Boot takes about 100 s on this database. #reconcileWaitingRunsAtBoot (SELECT id, account_id FROM runs WHERE status='waiting' AND not_before IS NULL) alone took 67 s. The healthcheck start_period is 60 s, so a restart looks unhealthy even when it would recover.

    Synchronous SQLite on the request thread. Any slow query anywhere stalls every request. This is the second read-path saturation in a week, after the ListSessions busy subquery on 11-03.

    Watchtower only acts on new images. An unhealthy container is never restarted or reported.

Detection gap

The Docker healthcheck failed continuously from about 05:50 and nothing watched it. Watchtower ignores health. The alerting proposed after the sandbox incident (seed#1334 and infra#24) is still unmerged. Its "metrics missing" rule would have fired here, because /api/metrics runs on the same blocked thread, but its event-loop-lag gauge would not, since a blocked loop cannot report its own lag.

Time to detection was over 6 hours. The same host had a 48-hour unpaged outage on 2026-10-31 when its data disk filled.

Mitigation and remediation

Status of all three workstreams: in progress.

    Privacy-scan hotfix (branch hotfix/agents-privacy-scan). Persist privacy flags when events are appended, fail closed for sessions not yet scanned, backfill old sessions in the background, and never read transcripts on the read path. Notes: https://hyper.media/hm/z6MkiAKDcRSzQ4zPZfnJcS5HYx5MwgN6MU9foHihJGrhqNBj/notes/agents-privacy-scan-hotfix

    Boot time, indexes and healthcheck (branch fix/agents-boot-sweeps). Index the boot sweeps and align the healthcheck start period with real boot time. Notes: https://hyper.media/hm/z6MkiAKDcRSzQ4zPZfnJcS5HYx5MwgN6MU9foHihJGrhqNBj/notes/agents-boot-time

    Database growth, retention and shrink runbook (branch fix/agents-db-retention). Notes: https://hyper.media/hm/z6MkiAKDcRSzQ4zPZfnJcS5HYx5MwgN6MU9foHihJGrhqNBj/notes/agents-db-growth

Background plans from 2026-11-03: agents server responsiveness and agents database choice.

Deploy path. Merge to main, then run the GitHub Actions workflow "Hotfix - Agents Docker Image" (.github/workflows/hotfix-agents-image.yml, workflowdispatch from main). Watchtower on agentic deploys :latest within 5 minutes.

Lessons

    A per-request check must be O(1) in history. Anything that scans a transcript on the read path will eventually outgrow memory. Privacy and similar classification belong at write time.

    An in-memory cache is not a fix for an expensive read. If a restart empties it, the restart reproduces the outage.

    Synchronous SQLite on one thread turns any slow query into a full outage. Slow reads need to be bounded, moved off the request thread, or both.

    A failing healthcheck that nobody watches is no healthcheck. This is the third agents-host outage in two weeks found by a person, not an alert.

    Diagnostics can add load. A docker exec child survives the SSH timeout that wraps it. Kill docker exec children explicitly, and prefer docker exec ... timeout N bun ... so the limit runs inside the container.

Action items

    [ ] Ship the privacy-scan hotfix: flags persisted at append time, fail closed, background backfill, no transcript reads on the read path. Owner: agents team.

    [ ] Ship boot-sweep indexes and a healthcheck start_period that matches real boot time. Owner: agents team.

    [ ] Add retention for large tool results and session files, and run the shrink runbook on agents-stable. Owner: agents team.

    [ ] Alert when any container on agentic is not healthy for more than 5 minutes, through Alloy/Grafana to Telegram or Discord. Owner: agents team.

    [ ] Merge the alerting in seed#1334 and infra#24, including the critical "metrics missing" rule. Owner: agents team.

    [ ] Audit the read path for other queries whose cost grows with history, and add a slow-statement budget to tests. Owner: agents team.

    [ ] Track database size per environment and alert on growth rate and on size against container memory. Owner: agents team.

    [ ] Wrap host diagnostics in an in-container timeout, and document it in the ops runbook. Owner: agents team.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime