Daemon operations
This page shows how to find out why a Seed daemon is slow or uses too much memory, with the read pool, the SQLite page caches, the debug pages and the flag that limits the heap.

The daemon is the program that stores, indexes and syncs Seed content. This page is for the person who runs one, on a site or on their own machine, and finds it slow or large.

The first thing to check is the daemon's /debug/sqlite page, on its HTTP port. It shows both things that go wrong under load: the SQLite read pool has no free connection, or the memory in use is far larger than the Go heap. This page explains each, names the debug page to open and the setting to change. The daemon page lists the flags and the routes named here.

Load and the read pool

Every read request takes a connection from the SQLite read pool before it runs. The pool has max(NumCPU, 12) connections, set in storage.go. A query that holds a connection for seconds takes that share of the node's read capacity away from every other caller. So a call that needs only microseconds of CPU can wait many seconds for a connection.

When a node is at its limit, measure the cost of one call, not the number of requests:

    Take a CPU profile from /debug/pprof/profile?seconds=20. It names the request that holds the connections.

    Open /debug/sqlite. It shows the pool.

    Count the calls of each method over the same time. Together these give the cost of one call.

A client must not read every page of an expensive listing, such as ListCitations for a document that many others cite. The web app reads only one page of those.

A restart only clears the waiting requests for a few minutes. Call a fix proven only after a long period of real traffic, never after the quiet minutes that follow a restart.

Memory

Most of the memory the daemon uses is C heap, not Go heap. Every connection in the pool keeps its own SQLite page cache of up to 256 MiB (cache_size in sqlite.go). The same limit applies to the in-memory sort buffer of each running ORDER BY or GROUP BY. SQLite frees that memory quickly: in WAL mode a reader drops its whole cache when it starts a read after a commit, and a sort buffer goes away with its statement.

The problem is what happens next. glibc, the C library, does not give the freed memory back to the kernel:

    It keeps freed pieces in the arena they came from, and only trims the top of each heap.

    Calls into C run on whichever thread is free, so the pieces spread over dozens of arenas.

    So the process grows to the sum of the highest point of every arena.

On Linux the daemon therefore calls malloc_trim(0) once a minute. That gives whole free pages from every arena back to the kernel and touches nothing that is still in use.

Two debug pages show where the memory is:

    /debug/sqlite shows SQLite's heap, its highest point, and every connection's page cache with hit and miss counts.

    /debug/malloc shows the memory the process holds, glibc's totals, the record of trims, and the report for each arena.

-storage.sqlite-soft-heap-limit limits the total in use. Above it, page caches reuse their oldest pages and do not grow. These pages and this flag are for one case: the system kills a daemon for lack of memory at many gigabytes, while its Go heap is under 1 GB.

The /debug/* pages follow the same access rules as the rest of the daemon's API.

See also

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Connections0
Linked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentLinked document · Outgoing linkLinked documentDaemon operationsDaemon operations
Citations 0Children & links 7

Loading child documents…

Connections show where other documents or comments link to this document.

Related documents

Loading connections…