Agents server boot time (2026-11-10)
Why agents-stable took ~100 s to start listening (a 67 s planner skip-scan in the waiting-runs reconcile), the fix (seed PR 1337), the migration cost on production, and how to verify it.

Fix: seed#1337, part of seed#1335. Incident: 2026-11-10 agents-stable privacy-scan wedge. The main wedge was the read-path privacy scans, fixed separately on hotfix/agents-privacy-scan. This note covers the boot.

What happened

When agents-stable restarted at 12:09 UTC, about 100 s passed before Agents server running appeared in the log. During that time the port was not bound at all. The Docker healthcheck (start-period 60s, interval 30s, retries 3) therefore marked the container unhealthy before the server could answer anything.

The slow-statement log showed these statements:

Statement

Time

Where

SELECT id, account_id FROM runs WHERE status = 'waiting' AND not_before IS NULL

67,169 ms

#reconcileWaitingRunsAtBoot, in the Service constructor, before listen

trigger_firings JOIN sessions

2.8 s

GetAgentTrigger, request path right after boot

SELECT COUNT(*) FROM sessions WHERE account_id = ? AND agent_id = ? AND parent_session_id IS NULL

1.7 s

GetAgent, request path

Why the waiting-runs select took 67 s

The query should have been a tight range on runs_dispatch (status, queue, not_before, created_at), because status is the index's first column. EXPLAIN on production showed something else: runs_dispatch (ANY(status) AND ANY(queue) AND not_before=?). That is a skip-scan, and a skip-scan drops the status equality.

The cause is in sqlite_stat1, which keeps one average row count per index prefix:

    status has a handful of values spread over every run ever made. The stats therefore say each status has tens of thousands of rows. waiting actually has a few dozen.

    not_before is NULL on almost every row, but distinct wherever it is set. The stats therefore say each not_before value has about 2 rows. The giant NULL bucket is hidden in that average.

From those numbers the planner decided that seeking not_before = NULL under every (status, queue) pair was cheaper than walking status = 'waiting'. That plan visits every NULL-not_before row of every status. For each row it does a lookup in runs to read account_id. runs is a WITHOUT ROWID table whose rows carry their input, output and wait blobs, so each lookup is expensive. On a cold, slow disk that cost 67 s.

Reproduced in a test and on a synthetic file (300k runs with 3 KB inputs, 1.5 GB, laptop SSD, prod-shaped stats):

Plan

Time

SEARCH runs USING INDEX runs_dispatch (ANY(status) AND ANY(queue) AND not_before=?)

11,586 ms

SEARCH runs USING COVERING INDEX runs_live (status=? AND not_before=?)

0 ms

Under the same statistics, the dispatch-tick lookups also left the status equality behind and fell back to skip-scans of runs_by_agent_status. These were #wakeDueTimers, #hasFutureWork, #hasDispatchableWork and the boot sweepAtBoot.

Two other things also made boot slow:

    The reconcile's own work. After the select it loads each waiting run in full (input and wait blobs) and its children. It also decoded the parent's whole transcript once per pending tool call.

    An unbounded ANALYZE runs / ANALYZE sessions on every boot (analyzePlannerTables). It walks the whole runs b-tree. It goes through db.run, so the slow-statement log never sees it.

The fix (PR #1337)


    Indexes (one prepended migration, mirrored in the baseline schema):

      runs_live ON runs (status, not_before, account_id) WHERE status = 'queued' OR status = 'claimed' OR status = 'running' OR status = 'waiting'. A query's status = 'x' implies one arm of the OR, so every equality lookup on a live status can use it. The index holds only unfinished runs (dozens), so ANALYZE gives the planner accurate numbers for it. The boot select becomes a covering seek. sweepAtBoot was rewritten from IN (...) to OR so that it qualifies too. Tests assert these plans under prod-shaped statistics, and also assert the old skip-scan when the index is missing.

      sessions_top_by_account_agent ON sessions (account_id, agent_id, parent_session_id) lets GetAgent's COUNT be answered from the index alone. The existing sessions_top_by_agent has no account column.

    Listen first. main binds the port before it opens the database:

      Health paths answer 503 {"status": "starting", "phase": …, "reason": …} with Retry-After: 5, and every other path answers 503.

      main yields to the event loop between boot steps, then swaps in the real routes and the WebSocket with server.reload.

      Each step logs [agents/boot] step finished {step, ms}, and Agents server running carries bootMs.

    Deferred, chunked reconcile. Service.recoverAfterBoot() runs the parked-parent reconcile and receipt pruning after the server serves:

      It takes the id list from runs_live.

      It re-reads each run before acting on it and skips anything no longer waiting.

      It yields a macrotask between runs.

      /api/health reports it as bootRecovery.

    What stays before listen. RunQueue.sweepAtBoot requeues every claimed/running row, so it has to run before this process claims anything. The effect-outbox sweep also stays. Both are single indexed statements now.

    Healthcheck. The agents/Dockerfile HEALTHCHECK --start-period goes from 60 s to 600 s. The SeedInfra agentic stack (seed_infra/agentic/main.tf, local checkout 02bde99) declares no healthcheck for agents-stable, so the image's healthcheck is the one in force and no SeedInfra PR is needed.

Exactly-once. The reconcile closes the crash window between a child's terminal commit and its parent's wait resolution. Moving it after listen keeps that guarantee:

    It only touches waiting runs, and the queue never claims those.

    Each step is synchronous, so nothing interleaves inside it.

    It replays the same idempotent path the live finalizer uses. The tool_result append is guarded by a transcript check, and resolveChildWait checks the status and the wait set. A live resolution or a cancel that lands between steps shows up on the next fresh read; a test covers both.

    The cost of moving it is that a parent orphaned by a crash resumes a few seconds after listen instead of before it.

Migration cost on production

Production statistics (read from sqlite_stat1 on 2026-11-10, no scans):

Index

sqlite_stat1

What it says

runs

61,091 rows


runs_dispatch

61091 15273 8728 47 1

About 4 distinct statuses (15,273 rows each on average), about 7 (status, queue) pairs, and 47 rows per not_before value

runs_by_agent_status

61091 3216 1528


sessions

22,645 rows


sessions_top_by_agent

22645 985 8 1 1


dbstat is not compiled in, so the table size in bytes is unknown.

Loaded into the planner as is, these exact rows reproduce the production plan: SEARCH runs USING INDEX runs_dispatch (ANY(status) AND ANY(queue) AND not_before=?). With runs_live present, the plan becomes SEARCH runs USING COVERING INDEX runs_live (status=? AND not_before=?), and the session COUNT uses sessions_top_by_account_agent as a covering index.

Index

Cost

runs_live

CREATE INDEX reads every runs row. status and not_before sit after input_cbor in the record, so rows whose input overflowed have their overflow chains read too. That is about one full read of the runs table, inside the migration transaction, holding the writer before listen. The index itself is tiny.

sessions_top_by_account_agent

22,645 rows on columns that precede the blobs. Well under a few seconds.

For runs_live:

    On the 1.5 GB synthetic file (300k runs) it took 0.5 s warm.

    Production has 61,091 runs, so the build reads at most a few GB of rows and overflow pages. On the slow disk I expect roughly 10–40 s.

    The 2026-11-03 runs_by_agent_status migration ran the same scan without incident.

    It is a one-time cost, and the 600 s start period covers it. During the build, health answers nothing, because the loop is held.

Limits

    /api/health and the loop-lag gauge cannot observe a fully blocked loop. A request that arrives during a synchronous step, such as a migration, ANALYZE, or the constructor sweeps, is answered only when that step ends. The starting answer helps because the socket is now bound before the database work and main yields between steps. It does not make a long synchronous step interruptible.

    The boot ANALYZE remains unbounded. I tried PRAGMA analysis_limit, but it samples index prefixes and skews the per-session and per-account averages. That is the kind of distortion that caused the 2026-09-23 plan regression, so I left it out until there is a measurement. Its time now shows in the open database boot step.

Verify after deploy

    docker logs agents-stable should show:

      [agents/boot] step finished lines with their ms;

      Agents server running … {bootMs} within seconds (plus the one-time index build on the first boot);

      [agents/boot] recovery finished {ms, parkedRunsChecked, …}.

    During boot, curl -s localhost:3050/api/health answers 503 with "status":"starting" and a phase. Afterwards it answers 200 with bootRecovery.state == "done".

    docker inspect --format '{{.State.Health.Status}}' agents-stable should read healthy, not unhealthy, after a restart.

    Read-only in the container: EXPLAIN QUERY PLAN SELECT id, account_id FROM runs WHERE status = 'waiting' AND not_before IS NULL should report COVERING INDEX runs_live.

    No [agents/sql] slow statement lines for the waiting-runs select or the sessions COUNT.

Follow-ups

    #sessionHasToolResult decodes a whole transcript on every child finalization. That is the same read-path pattern as the privacy scan. A per-session tool-result index or column would remove it.

    GetAgentTrigger joins every firing of a trigger to sessions with no LIMIT (2.8 s). It needs a LIMIT or a covering trigger_firings (account_id, trigger_id, created_at DESC, session_id).

    Make the boot ANALYZE incremental (PRAGMA optimize-style), or skip it when the stats are fresh.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime