An agent reaches the web in two ways. The read verb takes an https:// address and returns the page as markdown. The web_search callable tool finds pages. For example, read https://example.com/post returns the text of that post.
Both run on software the operator hosts. Neither uses an API key from another company. This page covers the order in which a web address is tried, the readers behind it, and the search backend. Operators set up the backends as described in Web research backends.
Hypermedia first
An https:// address is tried as Hypermedia first, because a Seed site serves documents at web addresses. The web reader is used only in two cases: the resolver says the URL is not Hypermedia, or the resource is not found (404).
Every other failure is reported as a failure: a short-lived daemon error, a result that is too large, a network error. So scraped page HTML never takes the place of a document's real content without anyone noticing (api-service.ts).
The reader chain
executeWebRead (apps/src/web-tools.ts) tries the cheapest reader first. It returns the first result that has 200 characters or more:
MediaWiki API for URLs that look like a wiki. It asks the host once through api.php?meta=siteinfo (the answer is cached per host) and fetches Parsoid HTML.
Static extraction inside the server: a plain fetch, then Mozilla Readability on a linkedom DOM, then Turndown to markdown.
Crawl4AI (POST /md, with a secret token in the Authorization header) when SEED_APPS_CRAWLER_URL is set. This is the last resort, for pages that need JavaScript or that refuse bots.
options.raw: true skips the chain and returns the response body as it is (text content types only). Markdown is limited to 200 KiB and is cut on a byte boundary.
web_search
web_search asks a self-hosted SearXNG instance (GET /search?format=json). SearXNG has no index of its own. It passes the query to public search engines and merges their answers.
Search engines often limit requests from datacenter addresses. So executeWebSearch (web-tools.ts) looks at unresponsive_engines. When the first query returns nothing and some engines did not answer, it retries once with a fallback set of engines.
When no searxngUrl is configured, the tool throws, and the error becomes tool_result.error.
The code matches the registry contract exactly. timeRange goes in, and is sent to SearXNG as its own time_range query parameter. A partial boolean comes out when engines did not answer, and the markdown names those engines. A test checks both directions.
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime