Slow agent search: account-scoped semantic search takes minutes (2026-11-13)
A hybrid search scoped to one account took 289 s on hyper.media. The cause is a quadratic filter in sqlite-vec plus a badly planned scoping subquery. It reproduces every time.

What happened

In this Clerk session the agent ran one search tool call that took 289,462 ms. It started at 11:28:05 UTC on 2026-11-13 and returned at 11:32:55. The whole run took 7.6 minutes and 4.8 of them were this call.

The input was:

{ "query": "schema default", "pageSize": 10, "accountUid": "z6MkuBbsB1HbSNXLvJCRCrPhimY6g7tzhr4qvcYKPuSZzhno", "searchType": "hybrid", "includeBody": true }

Five minutes later the same agent ran a keyword search with the same account scope. It took 72 ms.

Conclusion

Any semantic or hybrid search that is scoped to one account is slow on hyper.media, and the time grows with the size of that account. It is a bug in the daemon query path. It was not caused by load, by the deploy at 10:45 UTC, or by the embedding indexer.

Two things add up to the 289 s:

    About 36 s: the SQL subquery that finds which rows belong to the account walks 3.1 million index rows instead of starting from the account.

    About 250 s: sqlite-vec checks every stored vector against the resulting list of 150,243 row ids with a plain linear scan.

Evidence

I ran the same searches against hyper.media from my laptop, through the same Search API the agent uses.

    Keyword, scoped to the account: 0.4 s

    Semantic, no account scope: 2.6 s

    Hybrid, no account scope: 2.9 s

    Semantic, scoped to the Clerk account: 293 s

    Semantic, scoped to the starlight account: 138 s

So the scoped semantic path is the slow one, and it is slow again on every try. The daemon was idle otherwise. GetInfo showed state ACTIVE with no embedding task running, and the daemon logs for those five minutes have no warnings about search.

Then I measured the parts on a read-only view of the prod database.

    The scoping subquery alone takes 36 s and returns 150,243 row ids for the Clerk account.

    The starlight account gives 55,739 row ids.

    The vector table holds 5,530 chunks of 1,024 slots, which is 5.66 million slots.

Take 36 s off both scoped timings and you get 257 s for 150,243 ids and 102 s for 55,739 ids. That is 1.7 to 1.8 s per thousand ids in both cases. The cost scales with the size of the id list, which matches the code below.

Cause 1: linear scan in sqlite-vec

The scoped query is qEmbeddingsSearchFiltered in backend/llm/embedding.go. It filters with v.fts_id IN (subquery). In the embeddings table fts_id is a metadata column, so sqlite-vec handles the filter itself.

The vendored sqlite-vec is v0.1.7-alpha.2 at backend/storage/dbext/sqlite-vec/sqlite-vec.c. Its integer IN filter (around line 6385) does this for every chunk:

for(int i = 0; i < size; i++) { for(size_t target_idx = 0; target_idx < aTarget->length; target_idx++) { if( ((i64*)aTarget->z)[target_idx] == array[i]) { bitmap_set(b, i, 1); break; } } }

Every slot that is not in the list walks the whole list. With 5.66 million slots and 150,243 ids that is up to 850 billion comparisons, on one core. The same file uses bsearch for rowid IN (...), but the metadata path does not.

The comment on the query says the IN filter is there "for better performance". That was true when the tables were small. The feature shipped in February 2026 (commit baed66903) and the index has since grown to 2.66 million vectors.

I did not take a CPU profile of the C code. The attribution comes from reading the source and from the two timings that scale with list size.

Cause 2: the scoping subquery starts from the wrong table

The subquery joins fts_index to structural_blobs to resources and filters on r.iri GLOB 'hm://<account>*'. SQLite plans it as a search of fts_index by type, which visits all 3.1 million title, document and comment rows and checks each one's resource. The database has no sqlite_stat1 table, so the planner has no statistics to pick a better order.

Only 3,904 resources match the account. I rewrote the subquery to start from resources, then go to structural_blobs by resource, then to fts_index by blob, with CROSS JOIN to pin the order. On the prod data it returns the same 150,243 ids in 0.19 s instead of 36 s. The indexes it needs already exist (structural_blobs_by_resource, fts_index_by_blob, the blob_links primary key).

Why nothing stopped it


    Hybrid search runs keyword and semantic side by side and waits for both. Keyword was done in under a second. The result waited five minutes for semantic.

    The agent's search tool has no timeout on its request to the Seed server (executeAgentServiceSearch in agents/src/api-service.ts).

    The daemon sets no deadline on the semantic query.

    The agent's tool defaults to hybrid, and agents scope by account often because the tracker and their own space are single accounts.

The web search box uses keyword search, so site visitors do not hit this. The desktop search does use the semantic types, so a scoped desktop search can hit it against a large local index.

Each slow search also holds one daemon read connection and one core for its whole duration. A few of them at once would hurt the prod daemon.

Recommended fixes

In order of value for effort:

    Fix the filter in sqlite-vec. Sort the IN list once when it is built and use bsearch in the filter, as the rowid path already does. This turns 850 billion comparisons into about 100 million. Check upstream first in case a newer release already does it.

    Rewrite the scoping subquery to start from resources. This removes the 36 s floor that every scoped semantic search pays today, even for a tiny account.

    Give the semantic half of hybrid search a deadline of a few seconds and fall back to the keyword results when it is missed. The fallback code already exists for semantic errors.

    Add a timeout to the agent search tool so one bad query cannot stall a run for minutes.

Fixes 1 and 2 together should bring the scoped search close to the unscoped 2.6 s. I have not built or tested them. Until they ship, agents can avoid the problem by using searchType: "keyword" whenever they pass accountUid.

A later option is to make scoping part of the vector table itself, for example a partition key per space, so a scoped search only reads that space's vectors.

Notes on the investigation

My two reproductions each kept one prod daemon core busy, for about seven minutes in total. Everything else was read-only. No code or config was changed.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Attributes
name
Slow agent search: account-scoped semantic search takes minutes (2026-11-13)
summary
A hybrid search scoped to one account took 289 s on hyper.media. The cause is a quadratic filter in sqlite-vec plus a badly planned scoping subquery. It reproduces every time.