Agents Server Responsiveness
Why the agents server keeps pegging its one Bun thread, what today's outage measured, and a systematic plan (isolation, budgets, lag SLO, push not refetch, hot DB in RAM, right-sized box) so it stops recurring.

The pattern we keep fixing one symptom at a time

Since September every "agents feel slow" or "server stopped answering" episode has been the same failure class with a different trigger:

PR

What it fixed

Class

#1078

GetAgent carried every session; list sessions separately, paginated

one expensive query × client fan-out

#1080

stop refetching GetAgent and ListSessions on every transcript event

client fan-out

#1081

transcript frames only to direct subscribers

fan-out

#1161

pin run indexes the planner missed, ANALYZE on open

one expensive query

#1173

session lists load on first request

one expensive query (and it introduced today's one)

today (2026-11-03)

busy EXISTS forced onto runs_dispatch, which has no agent column

one expensive query × client fan-out

The common cause is architectural: one JavaScript thread serves HTTP and WebSockets, runs every SQLite statement synchronously, evaluates every trigger, and executes every agent run (Pi loop, CBOR, durable commits). Anything on that thread that costs more than a few milliseconds, multiplied by any client fan-out, starves accept() and the whole server goes dark while it is still "working". The worker-isolated execution plan said this in its first paragraph; only its phase 1 (workflow VM in a worker, behind a flag that is off in production) shipped.

What today's outage measured

Production agents-stable (commit 795b11385, 2026-11-03 ~17:20 CET):

    Main Bun thread at 90% CPU in state R. /api/health timed out at 30 s. Caddy logged dial tcp 172.18.0.7:3050: i/o timeout: the loop never reached accept. Agent runs kept progressing.

    15 s sampling profile through the Bun inspector: ~70% of main-thread samples inside the ListSessions handler, ~48% inside the ListAgents SQL it runs to label rows, ~10% in GetAgent. Run execution, trigger checks and WebSocket sends together were under 20%.

    The busy column in agentActivityColumns is EXISTS (SELECT 1 FROM runs INDEXED BY runs_dispatch WHERE agent_id = ? AND status IN (...)). runs_dispatch is (status, queue, not_before, created_at), so every non-busy agent re-fetches every live run row. On the production file: 120–200 ms per ListAgents, 1.5 ms without busy, 0.2 ms with a covering runs(agent_id, status) index (validated on an in-memory copy of the table).

    The same subquery runs on every GetAgent and on every session-event emit (#agentActivity), and the desktop refetches every cached sessions list with refetchType: 'all' on each session-created. Triggers started 137 sessions in 10 minutes. A few clients × a few lists × 150 ms per event exceeded the event rate, so the backlog grew without bound.

    Process disk I/O was low (1.7 MB read in 10 s). This was CPU inside SQLite page traversal, not disk.

    Secondary load on the same thread: processActivityEvent writes UPDATE agent_triggers SET last_checked_at per trigger per activity (autocommit, fsync each) and dumps a multi-line object with a bare console.log per check: 5,234 dumps and 120k log lines in 10 minutes. One trigger (e382f6cf…) keeps creating sessions that fail dispatch with "Message content is too large".

The box is not huge. agentic is an OVH d2-8: 4 vCPU (Haswell) and 8 GB, shared by agents-stable, agents-staging, agents-dev, Crawl4AI, SearXNG, Caddy and the microVM sandboxes (123 PIDs in the stable container alone). No container has a CPU limit. The stable database is 36 GB (1.5 M session events, 1.26 M durable-file rows) against a 64 MB page cache and 512 MB mmap on an 8 GB host.

Our actual needs are modest: about 8 concurrent model runs, 10–20 session starts a minute, a handful of connected clients. Served correctly that is a fraction of one core for the API.

Principles

Each principle below closes one way the loop has gone dark. Together they make the next regression impossible to ship unnoticed, and bounded in blast radius if it ships anyway.

P1. The API process does no run work

Split the service into roles in the same image, selected by SEED_AGENTS_ROLE=api|runner|all (default all, so the desktop's embedded server and tests do not change):

    api: HTTP, WebSocket fan-out, signed envelope verification, SQLite reads and enqueue writes, ActivityMonitor, ScheduleMonitor, trigger evaluation. It never executes a run.

    runner (N processes, N = cores − 1 on a dedicated box): claims runs from the existing lease queue (lease_owner, lease_expires_at, expiry re-claim already exist and were built for this), executes them with the same Service code, writes session events and durable files directly.

Why processes over the effect-bridge worker in the existing plan: the worker route must proxy 40+ Service members through typed messages (its phases 3–4, rated "high", untouched since). A process split reuses Service wholesale; SQLite in WAL mode is designed for several processes, and the queue already has leases. What must change is short and enumerable:

    Singletons run only in api: ActivityMonitor, ScheduleMonitor, queue drains, run-wait wakeups.

    #emit (68 call sites) in a runner forwards to the api over a loopback channel (internal /internal/emit with a bearer token, batched, fire-and-forget) so WS subscribers still get hints and transcript frames. Fallback for a missed hint: api polls PRAGMA data_version every 250 ms and re-sends a list hint.

    Per-process in-memory state is partitioned, not shared: the exec sandbox cap and warm pool per runner (SEED_AGENTS_EXEC_MAX_CONCURRENT divided across runners), session-list signal coalescing per process (harmless duplicates).

    busy_timeout = 5000, every write transaction short, and no long-lived read transactions in api so WAL checkpoints can complete (the WAL was 150 MB today).

    Compose for stable: one api container and two runner containers with CPU limits; dev and staging stay all.

Keep the shipped workflow-VM worker and flip SEED_AGENTS_WORKFLOW_WORKER on in runners.

Acceptance: /api/health and ListSessions p95 under 50 ms while 8 runs and 3 sandboxes are active, measured by the lag gauge below.

P2. Every statement on the API path has a budget, enforced in CI

Add a production-shaped SQLite fixture generator (20k sessions, 52k runs across 30 agents and 100 accounts, 1.5 M session events of realistic size, 36k trigger firings, 5k event claims) and a perf test that:

    runs every read action and the emit-path helpers against it and fails any statement over 10 ms;

    runs EXPLAIN QUERY PLAN over every SQL string the stmt() cache sees and fails on a SCAN of sessions, runs, sessionevents, triggerfirings or sessiondurablefiles, or on a correlated subquery whose inner search is not a covering index;

    refuses any INDEXED BY whose index does not lead with the subquery's equality column.

This single test would have failed #1173 before merge and today's regression on the first run. Every past "one expensive query" fix becomes a fixture row, not a hotfix.

P3. Event-loop lag is a first-class metric with an SLO

    A 100 ms timer measures lag; /api/perf reports its histogram (p50, p95, max over the last minute) next to the existing provider and tool spans. Today /api/perf said nothing was wrong while health took 28 s.

    A slow-statement log: any SQLite call over 20 ms logs the SQL text and duration at warn, rate-limited.

    The container healthcheck fails when p95 lag exceeds 250 ms for a minute, so docker ps, Caddy and Watchtower see it, and a fleet-notify alert fires from the healthcheck.

    The inspector profile recipe used today is documented in operations so the next person needs ten minutes, not a session.

P4. The server pushes; clients never refetch everything

    Remove refetchType: 'all' on session-created. The hint already carries the session snapshot; insert it into cached lists and only refetch the list that is on screen.

    Server-side singleflight: identical read actions from the same account in flight share one result; ListSessions and ListAgents get a 1–2 s cache invalidated by session and run writes.

    ListSessions fetches only the agents it references, by id, instead of listing the account.

    Per-connection concurrency cap (4 in-flight actions) so one client cannot queue hundreds of requests behind a stall and make it worse.

P5. The hot database fits in RAM

36 GB on an 8 GB host means every cold page is a disk read on the loop. Targets: hot DB under 4 GB.

    Archive session events and durable files of sessions finished more than 30 days ago into per-session CBOR files under /data/agents/<session>/ (or an attached archive database); reads of archived sessions fall back transparently.

    Bound session_durable_files growth per session and reclaim replaced versions.

    One-time VACUUM INTO a fresh file during a maintenance window, then swap; add the index migration from today at the same time.

    Keep PRAGMA cache_size proportional to the container memory (25%).

P6. Right-size and isolate the box

    CPU limits per container (the multi-server plan's own phase 1, never applied: cpus=0 today): stable 3 cores, staging and dev 0.5 each, Crawl4AI 1.

    Move dev and staging off the production host, or move production stable to its own 4-vCPU box. Staging was at 31% CPU and Caddy at 12% during today's outage.

    Memory for stable from 4 GB to 6 GB once dev and staging are gone, so the page cache in P5 has room.

Sequence

When

Deliverable

Closes

Now (hotfix)

runs_by_agent_status (agent_id, status) migration, drop INDEXED BY, ListSessions fetches referenced agents only, debug-gate the per-trigger log, batch last_checked_at into one write per poll, fix or disable trigger e382f6cf…

today's outage

Week 1

P3 lag gauge, slow-statement log, healthcheck and alert; P2 fixture and CI gate; P4 client refetch and singleflight

detection and fan-out

Weeks 2–3

P1 runner role behind SEED_AGENTS_ROLE, emit forwarding, singleton placement, compose for stable (1 api + 2 runners); validate on staging under synthetic load, then production

isolation

Week 3–4

P5 archival and VACUUM; P6 CPU limits, stable on its own box

capacity

Deploy hotfixes through the existing Hotfix agents image workflow. Each step is independently shippable and reversible; the role split defaults to today's behaviour.

Definition of done

    /api/health p95 under 20 ms and ListSessions p95 under 50 ms during a synthetic load of 8 concurrent runs, 3 sandboxes and 20 session starts a minute on staging, held for 30 minutes.

    A regression that adds a 50 ms statement to a read action fails CI.

    A runner pegged at 100% CPU for a minute does not change the api's lag histogram.

    Operator can answer "is the server slow, and where" from /api/perf in one curl, and gets a Telegram alert before a user notices.

Decisions needed

    Process split (P1 as written) versus continuing the worker effect-bridge plan. Recommendation: processes.

    Whether stable moves to its own box or dev and staging move off agentic. Recommendation: move dev and staging; production keeps the data volume and address.

    Archive format for P5: per-session CBOR files in the data directory (simple, rsync-able) versus an attached archive SQLite. Recommendation: files.

Evidence and recipes

    Profile: docker exec agents-stable bun /tmp/prof2.js ws://127.0.0.1:9230/<id> 15, inspector id from docker logs agents-stable | grep ws://. WebKit protocol: ScriptProfiler.startTracking {includeSamples: true}, samples arrive in ScriptProfiler.trackingComplete as stackFrames; the bundle is minified, so map line:column into /app/main.js.

    Read-only query timing on the live file: new Database('/data/agents.sqlite', {readonly: true}) inside the container, with the server's cache pragmas.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

People