Startup and shutdown
This page gives the order in which an app environment starts, the background tasks it runs while it is up, and what it does when it is told to stop.

This page follows one process of the app environment from start to stop. It says what each start step does, which tasks run in the background afterwards, and how the server stops. The code is apps/src/main.ts.

The first thing to check when a server does not come up: search its log for [agents/boot] step finished. Each start step writes that line when it ends, so the last one tells you how far the server got. What the server answers while it starts is in endpoints and health.

Startup

The server starts in this order:

    config.create(config.parseArgs()) reads the settings. An invalid setting or an unknown flag stops the process here.

    The HTTP server opens its port. It answers 503 with starting until the real routes are in place.

    sqlite.open(cfg.dbPath) checks the database or creates it: migrations, the open settings and the planner statistics.

    If the schema does not match, the server stays up in schema-mismatch mode. It answers a JSON error on every route and goes no further.

    If the schema matches, the server creates Service. That runs the boot sweep of the run queue and of the effect outbox. Then the activity monitor, the schedule monitor and the other background tasks start, and the real routes replace the start routes.

    Once the server answers requests, it handles parked runs whose children finished, and removes old receipts, one small step at a time.

When the schema does not match, the log names the stored version and the expected version. Persistence describes the check, and troubleshooting says what to do.

Each step logs [agents/boot] step finished with how long it took. Then the startup log prints the address, the database path, the HM API and IPFS URLs, the knowledge base account, the web tools that are connected, and the SQLite settings in use.

Background tasks

These run for as long as the process lives:

    Activity monitor. Every SEED_APPS_ACTIVITY_POLL_INTERVAL_MS (5 seconds by default) it reads the HM activity feed for each account that has an enabled trigger that needs the feed. It offers each event to trigger matching and to runs that wait for an event. It never polls for an account with no such trigger. See triggers.

    Schedule monitor. On the same interval it looks at the enabled schedule triggers and records a firing for each time that is due.

    Trigger document task. Every minute it turns the trigger documents under mounted agents and running apps into triggers on this server. It follows their edits and removals.

    Mount name task. Every minute it renames mounted agents after their documents.

    Run queue dispatcher. Every second it takes queued runs and wakes timers that are due. See the run queue.

    Loop-lag gauge. A 100 ms timer that measures how late it fires. See diagnosing a stalled server.

    Exec canary. Off unless SEED_APPS_EXEC_CANARY_SECS is set. See the code execution backend.

    Retention task. See database retention.

The monitors act only on the triggers in their own database. So several servers that read the same feed, each with its own database, do not fire on each other's agents.

Shutdown

The server stops when it gets SIGINT or SIGTERM. It does this, in order:

    It ignores any further stop signal.

    It stops the background tasks: the loop-lag gauge, the exec canary, the activity monitor, the schedule monitor, the trigger document task, the mount name task, and the retention task. This step takes at most 2 seconds.

    It closes every WebSocket client with code 1001.

    It empties its list of clients.

    It stops the HTTP server. If that takes more than 3 seconds, it forces the stop.

    It waits for background runs in progress, for at most 5 seconds, so one stuck session cannot hold up the stop. Then it stops the run queue's timers.

    It closes the SQLite database.

    It exits.

A run that was cut off is not lost. The next start puts it back in the queue; see the run queue.

See also

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

People