Agents privacy-scan hotfix (2026-11-10)
Reader privacy of agent transcripts is now stored as data and maintained on append, so API responses and WebSocket frames never scan a transcript. Unscanned history fails closed and a throttled backfill settles it.

What broke

On 2026-11-10 agents-stable accepted TCP connections and answered nothing. /api/health and /api/perf timed out after 30 to 60 s. The main Bun thread sat in D state, and the data disk ran at 98% utilisation doing about 9k random reads a second across the 46 GB /data/agents.sqlite, mostly overflow pages of large tool results. The most frequent slow statement read seq, event_cbor from session_events for a whole session, issued by BrowserPrivacy.sessionIsPrivate.

BrowserPrivacy.forViewer runs on every API response and on every collaborator or public WebSocket frame. The first time a non-owner touched a session or run, it decoded that transcript's entire history to decide whether it was private (did it use the browser or private memory, or read from a thread that did). It kept the answer only in memory, so after a restart it scanned every visible transcript again, and the event loop almost never got a turn. Restarting re-wedged the server within two minutes.

Incident report: agents-stable privacy-scan wedge. Tracking issue: seed#1335. Fix: seed#1336.

The fix

Privacy is stored as data

A new table, transcript_privacy, holds one row per session or run:

column

meaning

kind, id

session or run, and its id (primary key)

scanned_seq

last event or journal sequence folded into the row

private

1 once any scanned entry carried browser or private-memory data

sessions_json, runs_json

sessions and runs the scanned tool calls read from, whose privacy this transcript inherits

It is added by a migration and mirrored in the baseline schema. Nothing is backfilled in the migration itself, so it is instant even on the production file.

Write path

recordSessionEvent and recordRunEntry already saw every appended event and journal entry before fan-out. They now update the row. A contiguous append (seq = scanned_seq + 1) is folded in and advances scanned_seq. After a gap, which means history from before the row existed, a private append still marks the transcript private. Any other append queues the transcript for the backfill.

Read path

sessionIsPrivate and runIsPrivate now do primary-key lookups only: the transcript's row, MAX(seq) from the (id, seq) key (no body is read), and the parent and continuation lookups that were already indexed. Inherited privacy is resolved through the same lookups on the dependency rows. Definite private answers are cached in memory. No response or WebSocket path reads a session_events or run_journal body any more. A regression test records every statement executed during non-owner forViewer calls (cold, after backfill, and after a simulated restart) and asserts that none touches event_cbor or entry_cbor.

Fail-closed window

A transcript whose row is missing, or whose row is behind its last sequence, is pending. That covers every transcript written before this deploy, plus the rare case of a process dying between an append and its flag write. Pending transcripts are treated as private for everyone except the owner:

    Owners see everything, unchanged.

    Readers, collaborators and public viewers see a pending transcript redacted, not hidden. It still appears in session lists, but its title, description, events, run journal and partial-text frames are replaced by the same "Private content is only available to the agent owner" stub that browser sessions use.

    One place hides instead of redacting: the agent-facing thread: listing omits private threads from a non-owner run, so it omits pending threads too until they are scanned.

The window closes per transcript as soon as the backfill scans it. Transcripts nobody opens are never scanned, which is fine because they are never decided either.

Backfill

Reading a pending transcript queues it. A background timer then:

    serves the most recently requested transcripts first, and also queues their unscanned dependencies;

    works in 20 ms slices with 50 ms pauses, reading one event or journal body at a time and persisting progress after each slice, so it can stop mid-transcript and resume;

    drops to a 5 ms slice once a second while event-loop lag is high (p95 over 50 ms or last sample over 100 ms), so it cannot recreate the stall it replaces;

    logs at most once a minute ([agents/privacy] transcript privacy backfill {scanned, queued}).

Progress lives in the table, so a restart never rescans anything already scanned.

Agent runtime

An agent turn reads its own session's whole log anyway. Its three privacy checks (turn-state redaction, the durable-conversation append decision, and #piMessages redaction) use scanSessionIsPrivate, which scans that session and its sources synchronously, once, instead of failing closed. Without this, a delegated child or collaborator run on a pre-deploy session would have had redaction stubs in its own model context. That showed up as a failing test during the fix.

How to verify on prod

    After deploy, curl -s -m 5 https://<agents-stable>/api/health answers in well under 100 ms with status: ok. Watch loop.lag.p95 in the response settle.

    The slow-statement log (sql.slow warnings from statements.ts) no longer shows SELECT seq, event_cbor FROM session_events WHERE session_id = ? AND seq > ? ORDER BY seq or the run_journal equivalent. The backfill's statements read one row at a time (... ORDER BY seq LIMIT 1), and each should be fast.

    iostat -x 5 on the host: sdb utilisation drops from about 98% to background levels.

    The logs show [agents/privacy] transcript privacy backfill lines with queued trending to 0.

    As a public reader, open an old session of a public agent. It shows stubs at first, then its real content once the backfill has reached it (seconds).

Risks and follow-ups

    During the window, public and collaborator viewers of old sessions see stubs. This is deliberate: failing open would leak browser and private-memory content.

    The in-memory queue is capped at 10,000 entries. Older requests drop off and are queued again on their next read.

    transcript_privacy rows of deleted sessions are removed with the session. Run rows are never deleted, and neither are runs.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Comments