polite-crawler

polite-crawler

Fetch a source one time, keep it, and read it many times. polite-crawler is a polite web crawler for research and for AI agents. It obeys robots.txt, sends one request at a time to each site, and keeps each response on your disk. Then it answers each later question from a local full-text index, with no new request.

[npm] [CI] [license]

The site has interactive demos with the real code: https://mark1russell7.github.io/crawler/

• Polite. The crawler obeys robots.txt (RFC 9309), and the rules for Claude-User too when an assistant asks. It sends one request at a time to each host and obeys Retry-After. A challenge, a 401, a 402 or a 403 stops the host for a time.
• Fetch one time. Each response goes into a capture log and a store of bodies by hash. A repeated question reads the disk, so it costs no request.
• Read the cheap way. The crawler reads JSON-LD, the visible text and the links with no browser. It opens a headless browser only for a page that shows almost nothing without JavaScript.
• Cite exact text. Full-text search (SQLite FTS5) gives ranked passages. A quote check finds the exact span in the stored text, with the capture id to cite.
• An MCP server. The package polite-crawler-mcp gives Claude and other MCP clients nine tools. They fetch, find, read and verify pages, check field values, discover and crawl a site, compare versions and list the watched pages.
• Web archives. An export writes the stored captures as WARC or WACZ. Archive tools read them, for example warcio, pywb and ReplayWeb.page.
• Find and crawl a site. discover lists the pages of a site from its sitemaps and feeds. crawl follows the links of a site within a budget, at the pace of the site.
• Watch and compare. watch checks pages again on a schedule and tells what changed. diff compares two versions, and read takes at: the page as it was at a time.
• Archived copies, on request. When a site refuses a page or is down, fallback: "archive" gives a copy from the Wayback Machine or Common Crawl. The result marks the copy as archived, with its date.
• Signed requests. With an Ed25519 key, the crawler signs each request (Web Bot Auth). A site can then verify that a request comes from your crawler.
• Recipes. readVenue reads the opening hours of a venue from its site, and readProduct reads a product page. defineRecipe makes a recipe of your own.

Install

npm install polite-crawler

The package needs Node.js 22.19 or later. Older versions of Node.js 22 have no full-text search (FTS5) in their SQLite. For pages that need JavaScript, install the optional browser too:

npm install playwright-corenpx playwright-core install chromium-headless-shell

Without the browser, the crawler reads each page as plain HTML. It checks for the browser one time, so a page that needs JavaScript gets a note at once, with no wait.

Hybrid search (search by meaning and by words) is optional too. It uses a local model: transformers.js in the process (npm install @huggingface/transformers), or a local Ollama server. Hybrid search tells how.

The package is ESM only, and it includes its TypeScript declarations. It has these entry points:

The entry points that work in a browser import no Node.js module. The package loads no MCP SDK and no zod: the MCP server is the package polite-crawler-mcp.

Use it from the command line

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawlernpx polite-crawler find "robots exclusion"npx polite-crawler read "url:https://en.wikipedia.org/wiki/Web_crawler" --length 2000npx polite-crawler verify "url:https://en.wikipedia.org/wiki/Web_crawler" "robots.txt protocol"npx polite-crawler extract "url:https://en.wikipedia.org/wiki/Web_crawler" --field "name=Web crawler"

The store is in ~/.polite-crawler. Use --dir or POLITE_CRAWLER_DIR for a different folder. A path that starts with ~ is in your home folder, and a relative path starts at the working folder. Use --json for output that a program reads. npx polite-crawler --help shows all the options.

• --allow-private (or POLITE_CRAWLER_ALLOW_PRIVATE=1) lets the crawler fetch loopback, private and link-local addresses. By default, the crawler refuses them.
• read --user-provided reads a page that asked not to be AI input. Use it only for a URL that you gave yourself. The crawler records each such read.
• --max-age 0 fetches a page again, also when the store has a copy.
• fetch --archive-fallback gives an archived copy when a site refuses a page or is down. Archived copies after a refusal tells the rules.
• --sign-key <file> and --signature-agent <url> sign each request. key-directory --sign-key <file> prints the key directory to host. Signed requests tells how.
• find --rank fast makes a long question faster, and its results can be different. Fast ranking gives the measured difference.
• find --rank hybrid --embedder transformers (or --embedder ollama) also finds the pages with a similar meaning. embed --embedder <name> gives vectors to each stored page first. Hybrid search tells the options.
• reindex builds the index again. The documents of a recipe that the command cannot read (the recipes of another application) stay. The output counts them, and the captures that it skipped.
• export --out <file> writes the stored captures as a WARC or a WACZ file. Export to a web archive tells what goes into the file.
• extract <doc> --field name=value[:kind] checks the values that you propose for the fields of a stored page. Repeat --field for each field. --capture checks the text of one capture, and --at the version that was current at a time. Check field values against a page tells the statuses.

To find the pages of a site, or to fetch a part of a site, use discover and crawl:

npx polite-crawler discover https://example.com/npx polite-crawler crawl https://example.com/ --max-pages 50 --max-depth 2 --budget 2m

• discover prints each URL of the sitemaps and feeds of the site, with its lastmod. --max-urls limits the list (the default is 1000), and --feed <url> reads one more feed.
• crawl prints each page when its fetch ends, and then a summary. The options are --max-pages (default 100), --max-depth (default 3) and --budget (default 10m).
• crawl also takes --include and --exclude (patterns, which you can repeat), --sitemaps, --render and --max-age.
• discover and crawl take --allow-host <host> for more hosts. The first Ctrl+C stops a crawl, and the summary prints.

To read a page as it was, to compare two versions, or to watch pages, use read --at, diff and watch:

npx polite-crawler read "url:https://example.com/" --at 2026-10-01T12:00:00Znpx polite-crawler diff "url:https://example.com/" --from 2026-10-01 --to fetch.5d41402abc4bnpx polite-crawler watch add https://example.com/hours --every 6hnpx polite-crawler watch run --budget 2mnpx polite-crawler watch list

• read --at and verify --at use the version that was current at that time. Read a page as it was at a time gives the rules.
• diff prints the changed lines as a unified diff. A changed line marks its words with [-removed-] and {+added+}. --from and --to take a capture id or a time.
• watch add takes --every (1 hour or more, 24h by default), --confirm 1 or 2, --render, --scope and --label <name=value>.
• watch run takes --budget (default 10m) and --max-pages (default 100). It prints each check, then a summary. A scheduler of the operator starts it.
• daemon checks the watch list on a schedule, until you stop it. With --mcp-http, it also serves the MCP tools over local HTTP. The daemon tells how.

To read the hours of a venue or a product page, use venue and product. They use the recipes of the package:

npx polite-crawler venue https://forno-esempio.example/ --name "Forno Esempio" --pages 2npx polite-crawler venue https://forno-esempio.example/ --compare kept-week.jsonnpx polite-crawler product "https://bottega-campione.example/products/olio-esempio?variant=222"

• venue prints the week of the site and each reading with its URL, its rank and its quote. It also prints the days where the readings disagree, the menus, the phone, the pages and the notes.
• venue takes --name, --require-name, --address, --lat and --lon, --osm <value> and --pages <0-8>. Read the hours of a venue tells what each one does.
• Write a number below zero after =, for example --lon=-20.5. Else the command line reads it as an option.
• venue --compare <file> compares the week of the site with a week that you keep, as JSON. It prints same, changed, new or not read for each day. The file is a week, for example {"mon": [["07:00", "15:00"]], "sun": []}, or the from_site of venue --json.
• product prints the name, the brand, the SKU and the GTIN, and the price as money with the range of the offers. It also prints the availability, the variant, the seller, and a wall or a held-back page.
• The two commands take --render, --max-age, --budget, --user-provided, --archive-fallback and --archive-despite-robots. --on-behalf-of <agent> applies the robots.txt rules of an agent too, for example Claude-User.
• --json prints the VenueResult or the ProductResult of the library.

The command gives these exit codes:

Use it from Claude (MCP)

The MCP server is the package polite-crawler-mcp. It gives nine tools over the same store as the command line. Add it to Claude Code:

claude mcp add polite-crawler -- npx -y polite-crawler-mcp

Its README tells the tools, the envelope of the page text, the pages that refuse AI input and the options of the server.

Until 0.2.0, the server was the entry point polite-crawler/mcp and the command polite-crawler mcp. The command still starts the server in 0.2.x when polite-crawler-mcp is installed next to polite-crawler, with a note that it is deprecated. Without the package, it tells you to start npx polite-crawler-mcp, and it exits with 2. Version 0.3.0 removes the command.

Use it as a library

import { openCrawler } from "polite-crawler";
const crawler = openCrawler({ dir: "./crawl", contact: "[email protected]" });
const page = await crawler.fetch("https://example.com/", { maxAge: "24h" });console.log(page.outcome, page.title, page.capture, page.fetched_at, page.from_cache);
for (const hit of await crawler.find('"documentation examples"')) console.log(hit.doc, hit.snippet);
const read = await crawler.read(page.doc!, { length: 2000 });const check = await crawler.verify(page.doc!, "for use in documentation examples");
// later, after a new fetch: the quote is still in the capture that the citation namesconst cited = await crawler.verify(page.doc!, "for use in documentation examples", {  capture: check.capture!,});
await crawler.close();

• Each method gives a promise, find and verify too.
• A result of fetch gives capture, fetched_at and from_cache. A result of read gives the capture and the fetched_at of its text.
• verify gives the capture, the blob and the fetched_at of the text that holds the quote, with the positions in that text. stale is true when only an earlier capture holds the quote.
• read and verify take capture: they then use the text of that capture, not the current text. A citation of a capture thus stays true after the page changes.
• extract checks the values that you propose for the fields of a page, and gives the span and the capture of each value. Check field values against a page tells how.
• versions(doc) gives the versions of a document, the newest first.
• read and verify take at: the version that was current at that time. Read a page as it was at a time tells the rules.
• diff(doc, { from, to }) compares two versions. Compare two versions tells what it gives.
• watch is the watch list: add, remove, list and run. Watch pages tells the rules.
• find reads a query syntax. Text in double quotes is a phrase, and bake* is a prefix. The documents that hold each word come first. Case, accents and Unicode forms do not change a match, and Chinese, Japanese and Thai words match too.
• find takes rank. The default is exact, and fast makes a long question faster. Fast ranking tells the difference. hybrid also finds by meaning, with the option embedder of openCrawler. Hybrid search tells how.
• A page that asks not to be AI input is held back in each result. fetch gives no title, and read gives no title, no text and only the structural metadata. verify gives found: false, and find leaves the page out. Only userProvided lifts the hold, and the crawler logs it.
• The hold applies to each version too. When a later capture does not ask it, the current text is not held back, but the earlier versions that asked it stay held back. verify with no capture does not examine them, read and verify of their capture hold them back, and versions shows no title for them.

Fast ranking

The default ranking (rank: "exact") gives a BM25 score to each document that matches. A long question is the slowest search, because its common words match almost each document. rank: "fast" changes only the second part of a search, the documents that hold one word of the question:

• It leaves out each word that more than 20 percent of the documents hold (FAST_RANK_SHARE). When each word is that common, the rarest word stays.
• A document that holds only common words is not in that part of the results. The common words do not add to the score of that part.
• The documents that hold each word come first in the two modes, with the same rank.

The command line has find --rank fast, and the tool crawl_find has the parameter rank. DocIndex.rankWithin takes { rank: "fast" } too. It then leaves out each document that holds only common words.

The benchmark bench/find.mts measured the two modes in CI (the results). The index had 30,000 documents of 900 words, with a Zipf spread of the words. Each time is the median of 12 questions on Node.js 24:

• "Same top 10" is the share of the exact top 10 that fast also gives (overlap at 10). "Same order" is the Kendall tau of the documents that the two lists share. A value of 1 is the same order.
• In the corpus of the review, almost each word is in almost each document. There, fast was 2.2 times faster for 16 words, but its top 10 had only 43 percent of the exact top 10.
• For 16 words, rankWithin with fast gave 22 percent of the documents that exact gave.

Use fast only when the speed of a long question is more important than its exact rank. To measure again, start the workflow bench.yml.

The MCP server is the package polite-crawler-mcp:

import { openCrawler } from "polite-crawler";import { createMcpServer, startMcpServer } from "polite-crawler-mcp";
await startMcpServer({ dir: "./crawl" }); // standard input and output
const server = createMcpServer(openCrawler({ dir: "./crawl" }), { onBehalfOf: ["Claude-User"] });

createMcpServer takes these options: onBehalfOf (the default is ["Claude-User"]), allowUserProvided (the opt-in of the operator) and renderOff (the reason for a server with no browser).

The root entry point also exports the parts of the runtime, for an application that builds its own:

• createRuntime: the parts of a store, made in one place. openCrawler uses it. Give dir, or your own log, blobs and index (any ReadingIndex).
• catchUpIndex(runtime): the step of openCrawler at open. It rebuilds a stale index, and gives an index that is behind the log the captures that it missed. Use it after createRuntime.
• Crawler, CaptureLog, DocIndex, FileBlobStore, SqliteHostStateStore, Extractors, fetchAndIndex, fetchPage and reindex.
• readDocument, verifyQuote, findDocuments, isHeldBack, heldBackMeta and recordOverride: the read, the verification and the rule for a held-back page, over a runtime.
• extractDocument and checkFields: the check of field values over a runtime, and over a text and its metadata with no store.
• extractPage, typesOf, decodeBody and bodyEncoding: the page reader and the decoder of a body in its declared encoding.
• readPdf and isPdf: the PDF reader. PdfOptions sets limits on the pages, the characters, the time and the inflated bytes.
• fromText, fromSchemaOrg, fromCompact and describeWeek: the opening-hours reader. compareHours compares a reading with a week that you keep.
• parseRobots and decide: the robots.txt parser and matcher.
• RenderRefused: the error of a renderer that did not render a page, with the reason refused, download, scheme or browser. A crawler does not count it as a failure of the host. After the reason browser, the crawler renders no more pages.

The readers give more facts than before:

• PageExtract has canonical (the canonical URL on the same site), redirect (the target of a quick <meta http-equiv="refresh">), blocked (the name of a bot check or a firewall that sent its page in place of the content) and notes (what the reader left out).
• PageExtract also has feeds (the RSS and Atom feeds that the page links). extractPage always gives it. The field is optional, so a PageExtract of 0.1.1 stays correct.
• Hours has partial (the reader left out a rule or a line) and season (the season or the dates of the hours). The text reader reads English only.

The options of Crawler

CrawlerOptions has these options for the policy:

• egress: the addresses that the crawler fetches. By default, the crawler fetches only public addresses. Set allowPrivate: true for loopback, private and link-local addresses, or name hosts and ranges in allowHosts. With a dispatcher of your own, put egressLookup in its connection options.
• politeness: the limits of the policy, for each host or for some hosts (hosts: { "*.example.org": { pageIntervalMs: 10_000 } }). DEFAULT_POLICY has the defaults. A value that is not valid throws a RangeError.
• hostState: the store of the pace, the counts, the stops and the validator memo of each host (HostStateStore). The default keeps them in memory (MemoryHostStateStore). SqliteHostStateStore keeps them in a file that processes share.
• onEvent: a function that gets an event for each request, response, wait, stop and skip.
• obeyDefaultToken: a crawler with a token of your own also obeys the group of PoliteCrawler when its User-Agent links the bot page of polite-crawler. Site owners read that page. The stricter rules apply. Set false when you host a bot page of your own.
• maxRenders: the pages that the browser renders at one time. The default is 2, because each render holds a browser context. A render waits for a slot before the lease of its host, so the wait is not time of the host.
• signer: the signer of each request (Web Bot Auth). Signed requests tells its options.
• scope, labels and scopeRank: the labels of each record. One fetch of a robots.txt can serve requests of different scopes. Its record takes the labels of each request and the scope with the highest scopeRank, the most private one.

openCrawler takes egress, politeness and hostState too, and also clock, scope, onEvent, extractors and recipes (the recipes of an application). Its default hostState is the file hosts.sqlite in the store. Thus the processes on one store share the pace, the stops and the validator memo of each host.

FetchRequest.maxAgeMs can be "auto": the crawler then uses the freshness that the server gave (Cache-Control, Expires, or 10 percent of the age of Last-Modified). crawler.resetHosts(host?) removes the stops and the failure counts of one host or of each host, for a process that runs for a long time.

Which parts are stable

Before version 1.0.0, a minor version can change the API, and CHANGELOG.md tells what to change. Some parts change only for a strong reason. Other parts are the engine, and they change more often. The declaration of each part of the engine has the TSDoc tag @beta, so your editor shows it. The API review gives the tier of each export.

These parts are the stable API:

• openCrawler and the methods of PoliteCrawler: fetch, find, read, verify, versions, discover, crawl, reindex, close, diff, watch and extract. openCrawler gives an OpenedCrawler: a PoliteCrawler that has discover, crawl, diff, watch and extract. The five are optional in PoliteCrawler, so an object of your own that implements it stays correct.

• The options of openCrawler: OpenOptions and RuntimeOptions, which OpenOptions extends, with Extractor and ExtractedDoc for the option extractors and CrawlEvent and FetchOutcome for onEvent. The options log, blobs, index and renderer of RuntimeOptions and the member PoliteCrawler.runtime are the engine.

• The interfaces of the store on disk: CaptureRecord, CaptureReader, CaptureFilter, ScanOptions, DamagedLine and BlobStore.

• The options and the results of these methods: OpenOptions, FetchOptions, PageResult, Found, ReadResult, VerifyResult, DocVersion and ReindexResult. at of ReadOptions and VerifyOptions, and at and before_first of their results, are stable too.

• The options and the results of extract: ExtractOptions, ExtractResult, ExtractSummary, FieldClaim, FieldKind, FieldStatus, FieldResult, FieldSpan and FieldCandidate, with the names of the normalizations.

• The diff: diffTexts, DIFF_LIMITS, DiffOptions, DiffResult, DiffSide, DiffSummary, DiffHunk, DiffLine, WordPart, MetaChange, VersionChoice, DiffLimits, DiffInput and TextDiff.

• The time travel: isoTime and VersionAt.

• The watch list: WatchList, WatchOptions, WatchEntry, WatchPending, WatchKind, WatchRun, WatchRunOptions, WatchEvent, WatchSummary, WatchStop, WATCH_LIMITS and the file watch.sqlite (WATCH_SCHEMA).

• The daemon: startDaemon, scheduleWatches, changeCommand, readDaemonState, daemonStateFile, DAEMON_DEFAULTS, their options and results, and the state file (DAEMON_STATE_SCHEMA). The MCP seam of the daemon is the engine: DaemonMcpHooks, DaemonMcpHost, DaemonMcpOptions, DaemonMcpService, the options mcp and mcpHost, and Daemon.mcp. The MCP server of the daemon is mcpHttpHost of polite-crawler-mcp.

• The options and the results of discover and crawl: DiscoverOptions, DiscoverResult, DiscoveredUrl, DiscoverSource, DiscoverStop, CrawlOptions, CrawlRun, CrawlPage, CrawlSummary, CrawlStop, UrlPattern and ScopeOptions.

• The readers: polite-crawler/robots, /hours, /html, /classify, /politeness and /signals. The site and its tools use them in the browser.

• The command line, its exit codes, the store on disk and polite-crawler/record.schema.json.

• The export: exportWarc and exportWacz, their options (ArchiveOptions, ArchiveFilter, ArchiveSource), their results (WarcResult, WaczResult, ArchiveCounts) and the records of the archive.

• The archived fallback: fallback and archiveDespiteRobots of FetchOptions, the field archived of the results, and the types Archived and WebArchive.

• Signed requests: createWebBotAuthSigner, keyDirectory, verifySignature, KEY_DIRECTORY_PATH, KEY_DIRECTORY_TYPE, the option signer, and the types WebBotAuthOptions, WebBotAuthSigner, RequestSigner, KeyDirectory, DirectoryKey, KeyInput, SignatureAgentType, SignedRequest and SignatureCheck.

• The headers of a signature follow the draft. When the draft changes, the headers change with it, and CHANGELOG.md says so.

• Hybrid search: the option embedder of OpenOptions, the interface Embedder (the extension point) with EmbedCallOptions, and the adapters transformersEmbedder and ollamaEmbedder with their options (TransformersEmbedderOptions, OllamaEmbedderOptions, Pooling, EmbeddingModel).

• Also stable: HybridCrawler (the result of openCrawler), rank: "hybrid" (SearchRank, SearchOptions), embed, rankWithin with RankOptions and Ranked, and hybrid (HybridSearch). The three members are optional in PoliteCrawler, and OpenedCrawler does not change.

• The options and the results of hybrid search are stable too: HybridOptions, HybridFindOptions, EmbedOptions, EmbedResult, EmbedStop, Passage and Found.passage. So are HYBRID_DEFAULTS, OLLAMA_MODELS, TRANSFORMERS_MODELS, NO_EMBEDDER, NO_OLLAMA and NO_TRANSFORMERS.

• The recipe SDK: defineRecipe, runRecipe, readJson, the option recipes of openCrawler and createRuntime, and RESERVED_RECIPES. Its types are Recipe, RecipeDefinition, RecipeExtract, RecipeCapture, RecipeRequest, RecipeRequestOptions, RecipeRunOptions, RecipeResult, RecipeDoc and RecipeTarget.

• The recipes: readVenue, venueReadingsOf, compareHours, readProduct and productOf. Their options and results are stable too: VenueOptions, VenueResult, VenueReading, VenuePage, VenueLink, VenueLocation, VenueOsm, VenueSource, VenueRead, HoursComparison, ProductOptions, ProductResult, ProductRead, ProductVariant and Money.

The other exports of the root entry point are the engine. Examples are createRuntime, catchUpIndex, Crawler, CaptureLog, DocIndex, FileBlobStore, SqliteHostStateStore, Extractors, fetchAndIndex and makeRecord. An application can build on them. A minor version before 1.0.0 can change them, and its section in CHANGELOG.md says so.

The engine of the check of fields is checkFields, extractDocument, FIELD_KINDS, ExtractSource and FieldsChecked.

The engine of discovery and of the crawl is discoverSite, crawlSite, readSitemap, feedDate, DISCOVER_LIMITS, CRAWL_LIMITS, SITEMAP_LIMITS and DEADLINE_NOTE. It also includes Crawler.robotsAllow and the deadline of FetchRequest, RenderRequest and PageOptions.

The engine of the time travel, the diff and the watch list is versionAtTime, diffDocument, WatchStore, WatchStoreOptions, runWatch, WatchRunnerOptions and WatchRescheduleOptions. It also includes the optional method ReadingIndex.versionAt, DocIndex.versionAt and Crawler.hostStop, the optional method WatchList.reschedule and the option keepSchedule of WatchOptions. Each of these functions operates on a runtime of your own. The methods diff, read with at and watch of the crawler are the stable way.

The engine of hybrid search is createHybridSearch, HybridSearchOptions, VectorStore, VECTORS_SCHEMA and the types of the vector store. It also includes chunkText, chunkTokensOf, estimateTokens, CHUNK_LIMITS, fuseRanks, unitVector, isLoopback, FeatureExtractor, TransformersModule, the optional method ReadingIndex.digests and DocIndex.digests. The file vectors.sqlite is a projection, as the index is: it is not part of the stable store.

The engine of the recipes is registerRecipe, rankHoursLinks, namesVenue, locationMatches, pickProduct, pickVariant, readAmount, VENUE_LIMITS, PRODUCT_LIMITS, RECIPE_LIMITS and the steps of VenueRead and ProductRead. The words that rank a link or a label are the engine too, so a version can improve them.

The engine of the archived fallback and of the signatures is archiveExtractor, archivedDocId, and the fallback and archiveDespiteRobots of PageOptions. It also includes range, signalLines and originalHeaderPrefix of FetchRequest, ai_input of ExtractedDoc, sign of RenderOptions and signer of CrawlerOptions.

Version 0.2.0 removed five exports from the root entry point: checkedRenderer, renderedKey, pageDocId, renderAndIndex and pageExtractor. No documented use needed them. durationMs stays in the engine, because polite-crawler-mcp uses it.

Hybrid search

Hybrid search finds the pages that hold the words of a question, and also the pages with a similar meaning. It is opt-in: the default ranking stays exact BM25.

import { openCrawler, transformersEmbedder } from "polite-crawler";
const crawler = openCrawler({ dir: "./crawl", embedder: transformersEmbedder() });await crawler.embed({ budget: "10m" }); // vectors for each stored page
for (const hit of await crawler.find("a loaf with a dark crust", { rank: "hybrid" })) {  console.log(hit.doc, hit.score, hit.passage, hit.snippet);}

• rank: "hybrid" needs an embedder. Without one, the search rejects with NO_EMBEDDER. It does not change to BM25 in silence.
• crawler.rankWithin(q, docs, { rank: "hybrid" }) ranks a set of stored documents in the same way.
• openCrawler gives a HybridCrawler: an OpenedCrawler with hybrid, embed and rankWithin, and a find that takes rank: "hybrid".
• The command line has --embedder, find --rank hybrid and embed. The MCP tool crawl_find takes rank: "hybrid" when the server has an embedder.

npx polite-crawler embed --embedder ollama --budget 30mnpx polite-crawler find "a loaf with a dark crust" --rank hybrid --embedder ollama

transformers.js

This embedder uses the model in the process of the crawler. The package @huggingface/transformers is an optional peer, and the embedder loads it at its first text.

npm install @huggingface/transformers

import { transformersEmbedder } from "polite-crawler";
const embedder = transformersEmbedder({ model: "Xenova/bge-small-en-v1.5", quantized: true });

• The default model is Xenova/bge-small-en-v1.5: 384 dimensions and an input of 512 tokens. The 8-bit model (quantized: true, the default) is approximately 34 MB.
• The first use downloads the model from the Hugging Face Hub (huggingface.co). This download does not go through the polite crawler, so robots.txt, the pace and the capture log do not apply to it.
• transformers.js keeps the files in cacheDir. Its default is node_modules/@huggingface/transformers/.cache/. The command line uses <store>/models. With localOnly: true, the embedder uses only the cache and downloads nothing.
• The embedder pools the vectors of the tokens as the model expects: cls for BGE and mean for the other models. The option pooling changes it.

Ollama

This embedder sends the texts to a local Ollama server (POST /api/embed). The package has no dependency for it.

1. Install Ollama from https://ollama.com, and start it with ollama serve.
2. Pull a model: ollama pull nomic-embed-text.

import { ollamaEmbedder } from "polite-crawler";
const embedder = ollamaEmbedder({ model: "nomic-embed-text", baseUrl: "http://127.0.0.1:11434" });

• The embedder sends 32 texts in each request (batch). It adds the prefixes that nomic-embed-text and mxbai-embed-large expect.
• An error tells what to do: ollama serve when no server answers, and ollama pull <model> when the model is not there.
• The embedder accepts only a server on this computer (a loopback address). allowRemote: true (--allow-remote-ollama) permits a different computer.

An embedder of your own

An application can give an embedder of its own, as it can give a renderer:

import type { Embedder } from "polite-crawler";
const embedder: Embedder = {  id: "my-model-v1", // the model and its version: the store keeps it with each vector  dimensions: 384,  maxInputTokens: 512,  embed: async (texts, { signal, purpose }) => myModel.embed(texts, { signal, purpose }),};

purpose is "document" for the chunks of a page and "query" for a question. Give a new id when the vectors of the model change.

What goes where

• The embedder gets the text of the stored pages and the words of each question. With transformers.js, the text stays in the process. With Ollama, the text goes to the Ollama server.
• A page that asks not to be AI input does not go to the embedder, and no hybrid result gives it. The rule applies to each version, as for find.
• Only the download of a transformers.js model goes to the internet, to the Hugging Face Hub.

How a hybrid search ranks

1. Full-text search gives the BM25 candidates (candidates, 50 by default). A candidate whose text has no vectors gets them first.
2. The vector search gives the documents with the closest chunks (neighbors, 50 by default). It compares the vector of the question with each stored vector (a brute-force cosine).
3. Reciprocal rank fusion makes one list. Each document gets the sum of 1 / (k + rank) over the two lists, with the rank from 1. The constant k is fusionK, 60 by default.

• The snippet and the passage of each hit come from its best chunk. passage.start and passage.end are positions in the text of the capture of the hit, for read.
• A search gives vectors only to its candidates. Thus a page with the meaning and none of the words comes back only after embed gave it vectors.
• embed takes budget (10m by default), maxDocs and kind. Start it again to continue: it does not send a text with vectors again.
• find and rankWithin take candidates, neighbors, fusionK and signal. HYBRID_DEFAULTS has the defaults.

Chunks and storage

• A chunk is a window of whole sentences. It ends at the end of a paragraph when it can, and a longer sentence is cut at its spaces.
• A chunk has the input limit of the model in tokens with a margin of 10 percent, and 512 tokens at most. Two chunks next to each other share approximately 10 percent of their tokens. A text gives 64 chunks at most.
• The vectors go into vectors.sqlite in the store. The key of a row is the SHA-256 of the text, the number of the chunk and the id of the embedder. Thus a reindex keeps each vector, and the same text gets its vectors one time for each model.
• The file is a projection, as index.sqlite is. Delete it, and embed makes the vectors again.
• A vector takes 4 bytes for each dimension: 1.5 KiB for 384 dimensions and 3 KiB for 768. With the keys and the indexes of SQLite, a chunk of 384 dimensions took 2.4 KiB of the file in the benchmark.
• The first hybrid search reads the vectors of the embedder into memory. Thus the memory is approximately the size of the vectors.

The benchmark bench/find.mts --hybrid measured the time in CI (the results). A fake embedder of almost no cost gave the vectors, so the time of a real model comes on top. The index had 30,000 documents of 900 words, which gave 89,100 chunks of 384 dimensions. Each time is the median of 12 questions on Node.js 24:

• The brute-force cosine is linear in the number of chunks. It is a good fit up to approximately 100,000 chunks (approximately 30,000 pages of this size). A search then takes approximately 0.1 s, and the vectors take approximately 150 MiB of memory at 384 dimensions.
• One million chunks take approximately 1 s for each search and 1.5 GiB of memory. A larger store needs an index of the vectors, and the package does not have one.
• vectors.sqlite was 210 MiB for the 89,100 chunks.

Read a page as it was at a time

Each capture of a document is a version, and the index keeps each version. read and verify take at, a time. They then use the version that was current at that time.

const then = await crawler.read("url:https://example.com/", { at: "2026-10-01T12:00:00Z" });console.log(then.capture, then.fetched_at, then.before_first, then.text);
const check = await crawler.verify("url:https://example.com/", "open on Sundays", { at: new Date("2026-10-01") });

• at is an ISO 8601 text or a Date. A date with no time is the start of that day in UTC.
• The rules that choose the current version apply to the versions of that time. Thus a render stays current while the plain body under it stays the same, and the richer recipe wins.
• The result names the capture and the fetched_at of the version, and it gives at.
• When at is before the first capture, no version answers. before_first is true, and the note gives the time of the first capture. read then gives no text, and verify gives found: false.
• verify with at examines that version only, as verify with capture does. A match is not stale.
• Give capture or at, not the two. The two together are an error.
• The hold of a page that asked not to be AI input applies as before. A version that was held back at that time stays held back. When the document is held back at this time, each of its versions is held back too.
• The command line has read --at <time> and verify --at <time>. The MCP tools crawl_read and crawl_verify have at.

find has no at. The search index (FTS5) holds the current text of each document. A search at a time needs an index entry for each version, or a scan of each stored text. Neither is cheap, so the package does not give it.

Compare two versions

diff compares two versions of a stored document. It sends no request.

const diff = await crawler.diff("url:https://example.com/");console.log(diff.from.capture, diff.to.capture, diff.summary);
const since = await crawler.diff("url:https://example.com/", { from: { at: "2026-10-01" }, to: "fetch.5d41402abc4b" });

• from and to are a capture id, or { at }: the version that was current at that time. The default to is the newest version. The default from is the version before to of the same recipe and kind (plain or rendered). Thus a render is not compared with the plain body under it.
• hunks give the text as a line diff, in the form of a unified diff, with 2 lines of context. A changed line pairs a removed line with an added line. Its words give the words that stay, the added words and the removed words.
• meta gives the changes of the title, the URL and the metadata at their paths, for example meta.jsonld[0].openingHours. Each change has the JSON of the old and the new value, cut to 300 characters.
• summary counts the added, removed and changed lines, the added and removed words, and the changes of the metadata. same is true when the two versions are the same.
• The command line has diff <doc> [--from <version>] [--to <version>]. A version is a capture id or a time. The MCP tool crawl_diff gives the changes in the envelope.

The line diff uses the algorithm of Myers. Its time is linear when the change is small. DIFF_LIMITS gives the bounds:

• When a limit applies, the note tells which limit. exact is false when the line diff is not minimal or a changed line has no word diff. truncated is true when the result leaves out a part.
• steps counts the steps of the diff. The tests use it, because a count of steps does not change with the computer.
• limits in the options changes the bounds of one diff.
• When a side is held back, or the document is held back, the diff gives no text and no title. It compares only the structural metadata, and same is null.

Watch pages

A watch list keeps the pages that the crawler checks again on a schedule. The list is in watch.sqlite in the store.

await crawler.watch.add("https://example.com/hours", { every: "6h", labels: { lineage: "venue-17" } });
const run = crawler.watch.run({ budgetMs: 120_000, maxPages: 50 });for await (const event of run) console.log(event.kind, event.suspected, event.url, event.diff);console.log((await run.summary).due);
console.log(await crawler.watch.list());await crawler.watch.remove("https://example.com/hours");

add of a page on the list changes its options and keeps its state. It sets the next check one every after the last check, unless keepSchedule is true. A new entry is due at once.

run checks the entries that are due, the first due first. It gives an async iterable of events, and a summary. Its options are budgetMs (10 minutes by default), maxPages (100 by default) and signal.

• Each check is a fetch of the crawler with no stored copy (maxAge: 0). Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A stored ETag or Last-Modified date makes the check a conditional request.
• A run sends requests to 8 hosts at one time at most, and to each host one request at a time.
• When the budget ends before the request of a check, the entry stays due for the next run.
• When a check finds a stopped host (a 403, a challenge, or a Retry-After of more than 1 hour), the entry is due again at the end of the stop, or later.

Each event has the URL, the document, the capture of the check, the last confirmed capture (previous), the outcome and the kind of the check:

The confirm rule

One read can be a glitch: a page of an error, a test of the site, or a cache that is not current. Thus a change counts only when two checks see it:

1. The first check that sees a change gives it with suspected: true. The last confirmed version does not change.
2. The next check confirms the change when it sees the same new version. It gives confirmed: true, and the new version becomes the last confirmed version.
3. When the next check sees the last confirmed version again, it gives unchanged, and the note says that the change did not repeat.
4. When it sees a third version, that version is the new suspected change.

A confirmation needs a new capture: a stored copy does not confirm a change. gone and a page that comes back follow the same rule. With confirm: 1, each change is confirmed at once. A refused or an error check changes no state.

The diff of an event of the kind changed gives the counts of the diff from the last confirmed version. A page that asked not to be AI input gives no diff.

The time of the next check

Each check sets the next check of its entry one every later. An application with a schedule rule of its own can set a different time for one check:

const entry = await crawler.watch.reschedule("https://example.com/hours", { at: "2026-10-25T07:00:00Z" });await crawler.watch.add("https://example.com/hours", { every: "20h", keepSchedule: true });

• reschedule changes only the time of the next check. It gives the changed entry, or undefined when the list does not have the page.
• at is an ISO 8601 text or a Date. A time in the past makes the entry due at once.
• A run checks the due entries in the sequence of their times. Thus an earlier time also puts an entry before the other due entries.
• The time is not before one floor (1h, or watchFloor) after the last check, so a watch cannot hammer a site. An earlier time throws a RangeError.
• The check after it sets the next check one every later again. When a check is in progress, its end sets the time again.
• add with keepSchedule: true changes the options of a page on the list and keeps the time of its next check. Without it, add sets that time one every after the last check.
• Over a runtime of your own, WatchStore.reschedule does the same. The option job of runWatch puts a name in the captures of each check, as the option job of a recipe does.

The list in the store

• The list is the decision of the operator, not a projection of the capture log. Thus it has its own file, watch.sqlite. A deletion of the index or of hosts.sqlite does not lose it. The file opens at the first use of crawler.watch.
• Each step is one SQLite transaction (BEGIN IMMEDIATE) in WAL mode, and it waits for the lock of a different process (busy_timeout).
• A run claims each entry with a lease before its check. The lease holds until the end of the budget and 2 minutes more. Thus two processes that run at the same time check each entry one time. A process that stops holds its leases until they end.
• The list stays after a restart, with the state of each entry: the last confirmed capture, the suspected change and the counts.
• A scheduler of the operator (cron, a systemd timer or a job of an application) starts watch.run. Or the daemon starts it on a schedule: The daemon.
• The command line has watch add <url...>, watch remove <url...>, watch list and watch run. The MCP tool crawl_watch_list lists the entries.
• No MCP tool adds a watch. A watch commits future requests to a site, so it is an action of the operator, not of an assistant.

The daemon

The daemon is a local process that runs for a long time. It checks the watch list on a schedule, and it can serve the MCP tools over local HTTP.

npx polite-crawler watch add https://example.com/hours --every 6hnpx polite-crawler daemon --events ~/changes.jsonl --on-change "jq -c . >> ~/seen.jsonl"npx polite-crawler daemon status

import { startDaemon } from "polite-crawler";
const daemon = await startDaemon({  dir: "./crawl",  tick: "1m",  onChange: (event) => console.log(event.url, event.kind, event.capture),});console.log(daemon.status().next_due);await daemon.stop(); // or SIGINT and SIGTERM

The schedule

• Each tick is one watch run, with the budget and the maximum of checks of a tick. Thus each polite rule applies: robots.txt, the egress policy, one request at a time to each host, the stops of a host and Retry-After.
• The next tick comes when the first entry is due, and one tick (--tick, 1m by default) after the last tick at the latest.
• A random jitter from 0 to --jitter (10s by default) comes after it. Thus a check does not come before its time.
• When a tick stops at its budget (--tick-budget, 10m by default) or at its maximum (--tick-pages, 100 by default), the entries that stay due wait for the next tick.
• An entry that a different process holds waits for the end of its lease. Only one daemon operates on a store: a second daemon does not start.
• scheduleWatches(crawler.watch, options) gives the same schedule over the watch list of a crawler of your own.

The changes

A change is an event of watch run with confirmed: true: the last confirmed version of the page changed. The first version of a page is not a change.

• The daemon writes each change as one JSON line, to the standard output or to --events <file>. The library takes events (a file or a function) and onChange (a function).
• --on-change <command> starts your command for each change. The command gets the event as one JSON line on its standard input.
• The commands run one at a time, in the order of the changes. A command that runs for more than 1 minute stops. A command that fails becomes the last error of the status.

--on-change is a command of the operator. Obey these rules:

• The daemon starts the command in the shell of the system, as the user of the daemon, with the environment of the daemon. Give only a command that you wrote or trust.
• Do not take the command from a page, from an assistant or from a different user.
• The daemon gives the event only on the standard input, not in the command line. Thus no text of a page goes into a shell command.
• The event holds URLs and notes from sites. In your command, read the event as data, for example with jq. Do not give its text to eval or to a shell.

The stop

• At SIGINT or SIGTERM, the daemon starts no new check. The checks in progress get --grace (10s by default) to end and write their state.
• After the grace, the checks in progress abort, and the daemon releases their leases. The entries stay due for the next start. A second signal aborts the checks at once.
• The daemon stops its MCP server and cancels its tasks. It writes the state file, closes the store and exits with code 0.
• A daemon that stops with no clean stop (a crash, or a forced stop on Windows) keeps its leases until they end: the budget of the tick and 2 minutes.

The status

polite-crawler daemon status reads the state file <store>/daemon/state.json. It sends no request, and it opens no store.

• It shows the next due watches, the last tick, the next tick, the counts, the MCP endpoint, the tasks that run and the last error.
• It exits with code 0 when a daemon operates on the store, and with 1 when no daemon operates. --json prints the JSON of the state file.
• The daemon writes the state file after each tick and at each change of a task. The file is for the owner only.
• With --mcp-http, GET /status with the bearer token gives the same JSON.

MCP over HTTP

npm install polite-crawler polite-crawler-mcpnpx polite-crawler daemon --mcp-httpclaude mcp add --transport http polite-crawler http://127.0.0.1:9307/mcp --header "Authorization: Bearer $(cat ~/.polite-crawler/daemon/mcp-token)"

• --mcp-http serves the nine tools over the Streamable HTTP transport of MCP, at http://127.0.0.1:9307/mcp. --mcp-port and --mcp-host change the address.
• The MCP server is the package polite-crawler-mcp. The daemon finds it next to polite-crawler, not in the working folder. Without it, --mcp-http tells you to install it.
• The tools, the envelope, elicitation and --allow-user-provided are the same as for npx polite-crawler-mcp.
• crawl_site and crawl_discover are MCP tasks (the task primitive of the MCP specification 2025-11-25). The shared MCP host of @procedures/mcp-server gives the tasks.
• A client that asks for a task gets a task id at once. It polls the status (tasks/get), gets the result (tasks/result) and can cancel the task (tasks/cancel). A cancel stops the requests in progress.
• A client with no tasks gets the result in the call, with a progress notification for each page, as from npx polite-crawler-mcp.
• The status message of a task holds only counts, for example 3 of 10 done, and no text of a page.
• A task and its result stay for 1 hour. The server has 8 tasks in progress at most: a task above the limit fails at once, with the reason.
• Each session has its own tasks. When a session closes, its tasks stop.

The package polite-crawler-mcp has startMcpHttpServer(crawler, options). The daemon reaches it through a small seam, DaemonOptions.mcpHost, so a different MCP host can replace it.

The token

• Each request needs Authorization: Bearer <token>. The server compares the SHA-256 digests of the two tokens in constant time.
• By default, the daemon makes a new random token at each start. It writes the token to <store>/daemon/mcp-token, for the owner only where the system has file modes. On Windows, the folder of the store in your profile protects it.
• To keep one token across starts, write it to a file of your own and give --mcp-token-file <file>. The daemon reads the file and writes nothing. It warns when other users can read the file.

node -e "console.log(require('node:crypto').randomBytes(32).toString('base64url'))" > ~/.polite-crawler-tokenchmod 600 ~/.polite-crawler-tokennpx polite-crawler daemon --mcp-http --mcp-token-file ~/.polite-crawler-token

The address

• The server binds to 127.0.0.1 by default, so only your computer can reach it.
• On a loopback address, the server accepts only a loopback name in the Host header. An Origin header must name the same host. Thus a web page cannot reach the server by DNS rebinding.
• A host that is not loopback, for example --mcp-host 0.0.0.0, needs --allow-remote. Without it, the command line is wrong.

Warning: With --allow-remote, each computer that reaches the port can try the token. The server speaks plain HTTP, so a computer on the path can read the token and the pages. The tools also fetch pages from the address of your computer. Use --allow-remote only on a private network or behind a TLS proxy of your own. Do not open the port to the internet.

Start the daemon as a service

The package installs no service. These examples start the daemon at login, and they stop it with SIGTERM. Install the package one time (npm install -g polite-crawler), and change the paths.

On Linux, with a user service of systemd:

# ~/.config/systemd/user/polite-crawler.service[Unit]Description=polite-crawler daemonAfter=network-online.target
[Service]ExecStart=/usr/local/bin/polite-crawler daemon --events %h/.polite-crawler/changes.jsonlRestart=on-failureKillSignal=SIGTERMTimeoutStopSec=30
[Install]WantedBy=default.target

systemctl --user daemon-reloadsystemctl --user enable --now polite-crawlerjournalctl --user -u polite-crawler -fnpx polite-crawler daemon status

On macOS, with a launch agent of launchd:

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><!-- ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plist --><plist version="1.0"><dict>  <key>Label</key><string>io.github.polite-crawler.daemon</string>  <key>ProgramArguments</key>  <array>    <string>/usr/local/bin/polite-crawler</string>    <string>daemon</string>    <string>--events</string>    <string>/Users/you/.polite-crawler/changes.jsonl</string>  </array>  <key>RunAtLoad</key><true/>  <key>KeepAlive</key><dict><key>SuccessfulExit</key><false/></dict>  <key>StandardErrorPath</key><string>/Users/you/.polite-crawler/daemon.log</string></dict></plist>

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.github.polite-crawler.daemon.plistlaunchctl bootout gui/$(id -u)/io.github.polite-crawler.daemon

On Windows, with the Task Scheduler (PowerShell):

$action = New-ScheduledTaskAction -Execute "$env:APPDATA\npm\polite-crawler.cmd" `  -Argument "daemon --events `"$env:USERPROFILE\.polite-crawler\changes.jsonl`""$trigger = New-ScheduledTaskTrigger -AtLogOn# no time limit: the default limit of 3 days stops the daemon$settings = New-ScheduledTaskSettingsSet -ExecutionTimeLimit ([TimeSpan]::Zero) `  -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)Register-ScheduledTask -TaskName "polite-crawler daemon" -Action $action -Trigger $trigger -Settings $settings

The Task Scheduler stops a task with no signal. Thus the checks in progress stop at once, and their leases end after the budget of the tick and 2 minutes. daemon status then says that the daemon stopped with no clean stop.

Check field values against a page

A model or a person often reads a value from a page: a price, a date, opening hours, an author or a DOI. extract checks each proposed value against the stored copy, and it locates the value. It does not make values. It only checks and locates the values that you give, so each value has its proof.

const page = await crawler.fetch("https://example.com/focaccia");const result = await crawler.extract(page.doc!, {  price: { value: "4.50 EUR", kind: "money" },  since: { value: "2026-03-04", kind: "date" },  phone: { value: "+1 617 555 1234", kind: "phone" },  offer: { value: "4.50", kind: "json-ld:Product.offers.price" },  name: { value: "Focaccia" },});for (const [name, field] of Object.entries(result.fields)) {  console.log(name, field.status, field.span ?? field.path, field.normalization, field.capture);}

Each field gets one status:

The kinds are text (the default for a text), number (the default for a number), money, date, url, email, phone and json-ld:<path>. A kind that is not a kind throws a RangeError. The result also gives summary, the count of each status.

The normalizations

Each normalization has a name. A normalized field gives the names that it used, joined with +, for example thousands-separator+number-form.

An amount of money has a currency symbol or a code of ISO 4217 next to it, with one space or none. A code counts only in capital letters, so 15 try is not an amount.

The rule for ambiguity

A normalized match must have one meaning. The check does not guess:

• 1,234 and 1.234 can be 1234 or 1.234. Thus a number value that needs a normalization to match them is ambiguous. 1,234.5 and 12,50 have one meaning.
• 03/04/2026 can be 3 April or 4 March. 17/10/2026 and 10/17/2026 have one meaning.
• $ is the symbol of more than one currency, so $12 does not confirm USD 12. US$12 confirms it.
• March 4 has no year, so it does not confirm 2026-03-04.
• When the value matches two places with different meanings, the field is ambiguous, with each place. An example is two phone numbers that end in 555-1234.
• A value with two meanings, for example 03/04/2026, is ambiguous, unless the stored text holds its exact text.

An exact match is always verified: the stored text holds the text of the value, and the caller gives it its meaning.

Where the check looks

• The kinds of text read the text of the version, the text that read gives. The positions are positions in that text. When the reader cut the text, the check also reads the whole text, and the span then has in: "full_text".
• url also reads the stored links of the page, with the path links[3]. The store keeps the canonical form of each link.
• json-ld:<path> reads the JSON-LD of the page. The first part of the path names a schema.org type, and the path starts at each object of that type, at any depth. * starts at each top-level record.
• Each array on a JSON-LD path gives each of its values. The result gives the concrete path, for example jsonld[0].offers[1].price, and the raw value. It compares as a number, a date, a URL and a text, in that sequence.
• With no option, the check reads the current version. capture reads the version of one capture, and at reads the version that was current at a time. Give capture or at, not the two.

A page that asked not to be AI input gives held-back for each field, with no position and no capture. The result is the same for a value that is in the page and for a value that is not. Thus the check is not an oracle. userProvided lifts the hold for a URL that a person gave directly, as for read and verify. The crawler logs it.

The time of the check is linear in the length of the text, for a fixed set of fields. The tests count the steps of the readers on texts of different lengths.

The MCP tool crawl_extract gives the statuses and the positions in its JSON. The stored text, the readings and the excerpts come in the <untrusted-web-content> envelope. The command line has polite-crawler extract.

Read the hours of a venue

readVenue reads the opening hours of a venue (a restaurant, a shop or a museum) from its own site. It fetches a few pages, and it gives each reading with its source.

import { compareHours, openCrawler, readVenue } from "polite-crawler";
const crawler = openCrawler({ dir: "./crawl" });const venue = await readVenue(crawler, "https://forno-esempio.example/", {  name: "Forno Esempio",  osm: "Mo-Fr 07:00-15:00; Sa 08:00-13:00; Su off", // a value that you got from OpenStreetMap});console.log(venue.best?.hours, venue.best?.capture, venue.disagree);for (const reading of venue.readings) console.log(reading.from, reading.label, reading.hours, reading.url);
const change = compareHours(keptWeek, venue.from_site!.week);console.log(change.status, change.days); // changed [ 'sat' ]

The recipe reads these sources:

1. The home page: its JSON-LD (openingHoursSpecification and openingHours) and its text.
2. The pages that most likely hold hours or a menu. rankHoursLinks ranks the links of the home page by their words, for example "hours", "menu" and "visit". pages sets the number of these pages: 3 by default, and 8 at most.
3. An OpenStreetMap opening_hours value, when you give osm. The recipe sends no request to OpenStreetMap.

• Each page is a fetch of the crawler. Thus robots.txt, the pace, the stops, the egress policy and the capture log apply. A page that needs JavaScript gets the browser, as for fetch.
• Each reading has from, label, week, hours (the week as words), inferred, partial, season and raw. It names its url, doc and capture, and a reading of JSON-LD names its path.
• The raw of a reading of the text is a quote: verify(reading.doc, reading.raw, { capture: reading.capture }) finds it.
• rank orders the readings. JSON-LD is 0, a text under an hours heading is 1, OpenStreetMap is 2, and a text with no other label is 3. A different schedule (a brunch, a kitchen, a happy hour or an office) is 4.
• best is the first reading of the opening hours. from_site is the best reading of the site itself, with no guessed am or pm. A check for a change compares from_site.
• disagree names the days where two readings of the whole schedule (a rank of 2 or less) disagree.
• The result also gives menus (the menu links that are not on a review or social site), phone (from the JSON-LD) and captures.

• A location of a page matches when it has the same street number and a word of the street, or when it is within 150 m.
• moved names the new site when the home page landed on a different host.
• A page that asked not to be AI input gives no reading, and held_back is true.
• The recipe writes nothing but the capture log. It does not decide which reading is true. That decision belongs to your application.

venueReadingsOf(doc, { location }) reads one stored document with no request. Thus it also reads an earlier version:

import { venueReadingsOf, versionAtTime } from "polite-crawler";
const { version } = versionAtTime(crawler.runtime, "url:https://forno-esempio.example/", "2026-10-01T00:00:00Z");const then = venueReadingsOf(version!);

compareHours(kept, reading, { skipSplit }) compares a reading with the week that your application keeps. It is in polite-crawler/hours too.

• It gives status (same, changed or new), the changed days, the added days and the skipped days.
• It compares only the days that the two weeks state. A day that only the reading states is added, not a change.
• 24:00 and 00:00 are the same time, and 9:00 is 09:00. The sequence of the intervals of a day does not matter.
• With skipSplit, it does not compare a day with two or more intervals, for a store that keeps one interval for each day.
• One reading can be an error of the site. Confirm a change with a second reading, as a watch does.

The command line has polite-crawler venue, and venue --compare <file> does this comparison. Use it from the command line gives the options.

Read a product page

readProduct reads the schema.org Product data of a product page. Most shop systems write this data on each product page.

import { readProduct } from "polite-crawler";
const product = await readProduct(crawler, "https://bottega-campione.example/products/olio-esempio?variant=222");if (product.wall) console.log("the shop refused the crawler:", product.note);else console.log(product.name, product.price, product.price_to, product.availability, product.variant);// Olio Esempio { amount: 9.9, currency: 'EUR', raw: '9.90' } { amount: 18, ... } InStock { sku: 'OE-500', ... }

• The result gives name, brand, sku, gtin, price, price_to, availability, availabilities, variant, seller and image. It names the doc and the capture of the page.
• A price is { amount, currency, raw }, and the currency is an ISO 4217 code. € and £ count too, because each one names one currency. A price with no currency, or with $, is not money, and a note says so.
• price is the lowest price. price_to is the highest price, when the offers give a range (sizes, colors, variants). The prices are in the first currency of the offers.
• A list price and a struck-through price (priceType ListPrice or StrikethroughPrice) are not the price.
• variant is the variant that the URL names with the parameter variant, sku or gtin. The recipe reads its price alone.
• availability is a schema.org name, for example InStock. When the offers do not agree, it is InStock if one offer is in stock.
• The options are render, maxAge, onBehalfOf, signal, budget, scope, labels, job, userProvided, fallback and archiveDespiteRobots, as for readVenue.

A page often has more than one Product record: the related products, and the products that others bought. The recipe picks the product of the page:

1. The product whose url or @id is the page. An @id that is only a fragment (#product) does not count, because a related product can have one too.
2. Else the largest product with offers or variants.
3. Else the largest product. The recipe also reads the mainEntity of a WebPage or an ItemPage.

• The index keeps 30,000 characters of the JSON-LD of each page. A product with many variants can be larger, so the recipe reads the stored body again.
• When the plain page has no Product data, the recipe renders the page once, because some shops write that data with JavaScript. render: "never" stops this.
• A shop that refuses the crawler gives wall: true, with a note. The causes are a 401, a 402 or a 403, a challenge, a bot check, a stop of the host or robots.txt. The recipe does not try a different identity.
• A page with no Product data gives found: false. It is often a list of products.

readAmount reads an amount as a shop writes it. The last separator is the decimal point. One kind of separator before groups of three digits separates thousands, so 1.299 is 1299. A currency with three decimal digits (KWD) reads 1.299 as 1.299. A range, a negative amount and zero give no amount.

productOf(records, url) reads the JSON-LD records of a page with no request, so it also reads an earlier version. extract checks each value against the stored page:

const check = await crawler.extract(product.doc!, {  price: { value: product.variant!.price!.raw, kind: "json-ld:Product.offers.price" },});

The command line has polite-crawler product. It exits with 4 at a wall, and with 3 when it reads no Product data.

Writing a recipe

A recipe tells the crawler what to fetch and how to read it into documents. The venue and product recipes read pages with the page reader of the package. A recipe of your own reads a source that the page reader does not know, for example an API.

import { defineRecipe, openCrawler, readJson, runRecipe } from "polite-crawler";
interface Item {  readonly id: string;  readonly name: string;  readonly price: string;}
const menu = defineRecipe({  name: "forno-menu", // it goes into each capture record: do not change it  mode: "api", // a documented API: robots.txt does not apply, and the pace does  minInterval: "2s", // the documented pace of the API  maxAge: "6h",  fetch: (day: string) => `https://api.forno-esempio.example/menu?day=${encodeURIComponent(day)}`,  extract: (body, capture) => {    const { items } = readJson(body, capture) as { items: Item[] }; // a body that is not JSON throws    return items.map((item) => ({      doc: `menu:forno-esempio/${item.id}`,      kind: "menu-item",      title: item.name,      text: `${item.name}: ${item.price}`,      url: capture.url,      meta: { price: item.price },    }));  },});
const crawler = openCrawler({ dir: "./crawl", recipes: [menu] });const run = await runRecipe(crawler, menu, "mon");console.log(run.outcome, run.capture, run.docs.map((doc) => doc.doc));
await crawler.find("focaccia"); // the documents of the recipe are in the indexawait crawler.read("menu:forno-esempio/focaccia", { at: "2026-10-01T00:00:00Z" }); // each version stays

defineRecipe takes these values:

runRecipe(crawler, recipe, input, options) fetches the request of an input through the crawler, and reads its capture with the extractor.

• The result gives outcome, status, url, docs, capture, fetched_at, from_cache, held_back, error and note. The url shows each secret parameter as REDACTED.
• Each document gives doc, kind, title, url, meta, chars, capture and fetched_at. read gives its text. A held-back document shows no title and only its structural metadata.
• The options are maxAge, onBehalfOf, signal, scope, labels, job, budget, userProvided, fallback and archiveDespiteRobots.
• Give each recipe to openCrawler (or createRuntime) with recipes. Then reindex reads its captures from the start. runRecipe also registers a recipe the first time that it gets it.
• registerRecipe(runtime.extractors, recipe) registers a recipe in a runtime of your own. A different extractor with the same name throws an error.
• recipe.request(input, options) gives the request of the crawler, for fetchAndIndex in an application that builds its own steps.
• readJson(body, capture) reads a body as JSON, in the encoding that the response declares. A body that is not JSON throws an error that names the URL and the content type.
• runRecipe, readVenue and readProduct take a crawler or a runtime (createRuntime). The option onBehalfOf of openCrawler does not apply to them, so give onBehalfOf to each call.

The rules of an extractor

1. An extractor is pure. The same body and capture always give the same documents. Do not read the clock, a random number or a state from an earlier call.
2. An extractor sends no request. It gets the body of the capture. To read a second source, start a second recipe. Its fetch is then polite, and it has its capture.
3. The runtime reports an error, and the work continues. An extractor that throws gives no document. runRecipe gives the error in error, and reindex counts it in failed and errors. The fetch and the rebuild continue.
4. The SDK checks each document. Each document needs doc, kind, title, text, url and meta, and meta must be JSON. Two documents with one id, the prefix archive: and the kind archived throw an error. The archived fallback uses that prefix and that kind.
5. A held-back body stays held back. Set ai_input: "no" when the body asks not to be AI input. An extractor cannot lift a no of the capture.

The rules of the capture ids

• The crawler makes the capture id, not the extractor. The id is a hash of the source, the kind, the external id and the payload of the capture record.
• The payload holds the URL, the name of the recipe, the outcome, the hash of the body and the time of the fetch.
• Thus the name of a recipe is part of each capture id. Do not change the name after the first capture. With a new name, reindex finds no extractor for the old captures, and it keeps their documents as they are.
• Each version of a document names its capture. A citation names that capture, and read, verify and extract take capture to use that version.
• reindex reads each capture again with the extractor. A pure extractor thus gives the same versions with the same capture ids.
• A new version of an extractor changes the documents at the next reindex, not the capture ids. Keep it able to read the old captures.
• The scope and the labels of a fetch are not part of the capture id.

The documents, the time travel and the archived fallback

• Give each document an id with a prefix of your own, for example menu:. The page reader uses url:. A recipe that gives a url: id adds versions to the document of that page, and merge chooses the current version.
• Each capture of a recipe adds a version of each of its documents. read, verify and extract take at, and diff compares two versions.
• With fallback: "archive", a recipe of the mode page gets an archived copy after a refusal. The page reader reads the copy as the document archive:<url>, with archived.
• The extractor of a recipe reads only the live captures. Thus its documents do not mix a live answer and an old copy of a third party. A recipe of the mode api takes no fallback.
• When the extractor cannot read a stored copy, runRecipe asks the source one more time. Thus a bad answer (a page about maintenance) does not stay until maxAge ends.

Find the pages of a site

discover finds the pages of a site from its sitemaps and its feeds. It fetches the files that list the pages, not the pages.

const found = await crawler.discover("https://example.com/");for (const url of found.urls) console.log(url.url, url.lastmod, url.via, url.source);console.log(found.stopped, found.dropped);

Discovery reads these sources, in sequence:

1. The page at the URL that you give, for the RSS and Atom feeds that it links with <link rel="alternate">. With linkedFeeds: false, discovery does not fetch the page.
2. The feeds in feeds, and the sitemaps in sitemaps.
3. The Sitemap: lines of robots.txt. When robots.txt names no sitemap and sitemaps is empty, discovery reads /sitemap.xml.
4. The sitemaps of each sitemap index, the newest lastmod first.

Each URL of the result has its lastmod and changefreq when the file gives them. Its source is the file that listed it, and via tells if that file is a sitemap or a feed. The newest lastmod comes first.

• Discovery keeps a URL only when it is on the same site and robots.txt permits it. The rules for the crawler and for the agents of onBehalfOf apply. A host and its www. form are one site.
• allowHosts adds more hosts (*.example.org names the hosts under a name). sameSite: false keeps the URLs of each host.
• Discovery reads XML sitemaps, sitemap indexes, gzip sitemaps (.xml.gz), text sitemaps, RSS 2.0, RSS 1.0 and Atom.
• Each file is a fetch of the crawler, with the recipe sitemap or feed, and the capture log records it. A stored copy younger than maxAge (24 hours by default) answers with no request.
• sources tells what discovery did with each file. dropped counts the URLs that it did not keep, and stopped tells why it stopped.

These options limit a discovery. DISCOVER_LIMITS gives the defaults.

Discovery also reads robots.txt for 10 origins at most, and 5 feeds at most from the links of the page.

The sitemap reader (readSitemap) is strict, because a file from the web can be hostile:

• It expands no entity. It knows only the five entities of XML and the character references, and it ignores a DTD. Thus a "billion laughs" file gives no text.
• It stops at a depth of 32 elements, and at 50,000 entries (the limit of the sitemap protocol). A value of more than 4,096 characters is not valid.
• It reads the text one time and builds no tree, so its time is linear in the length of the file.
• It stops at the first error of the markup, and it keeps the entries before the error.

Crawl a site

crawl fetches the seeds, and then the links of each page on the same site, to a budget. A for await loop gets each page when its fetch ends.

const run = crawler.crawl("https://example.com/", { maxPages: 50, maxDepth: 2, budgetMs: 120_000 });for await (const page of run) console.log(page.depth, page.source, page.outcome, page.url);
const summary = await run.summary;console.log(summary.stopped, summary.reason, summary.fetched, summary.refused, summary.skipped);

• The seeds come first, then the pages of the sitemaps (the newest lastmod first), then the links. A lower depth comes before a higher depth.
• A pattern that starts with / matches the path and the query. A different pattern matches the whole URL. A * matches any text, and a $ at the end matches the end, as in robots.txt. A RegExp tests the whole URL.
• The scope and the patterns apply to the links and to the pages of the sitemaps, not to the seeds.
• The crawl follows only http(s) links, and it fetches each page one time. A fragment, a tracking parameter or a default port does not make a different page.
• The crawl does not fetch a URL that is the final URL or the canonical URL of a page that it fetched.
• The links come from the stored document of each page, 100 or fewer for each page. The crawl does not follow the links of a page that asked not to be AI input.

The summary counts the pages (fetched, from_cache, refused and failed) and the links that the crawl did not follow (skipped, with the reasons in skips). queued counts the URLs that waited at the stop. stopped and reason tell why the crawl stopped: done, max-pages, budget, cancelled, closed or error.

The crawl is polite

The frontier does not change a rule of the crawler, because each page is a fetch of the crawler:

• robots.txt applies to each page. A link that robots.txt refuses costs no request, and the summary counts it in refused.
• One request at a time goes to each host, at the pace of the host. By default, a host rests 3 seconds or more between two pages, so a crawl of one site gets approximately 20 pages in each minute.
• The frontier sends requests to 16 hosts at one time at most. The crawler also limits the hosts in progress (maxActive).
• The stops, the backoff, the egress policy, the redaction of secrets and the capture log apply as for fetch. The User-Agent stays the honest default.
• When the budget ends, no new request starts, and a request in progress ends normally. A request that waits for its host past the budget does not start. Its skip record says deadline reached.
• When signal aborts, or a break ends the loop, the crawl stops the requests in progress and ends.

The frontier keeps its state in memory, for one call. It keeps 10,000 URLs that wait and 100,000 URLs that it saw, at most (CRAWL_LIMITS). A later version can make the frontier resumable.

Archived copies after a refusal

When a site refuses a page, or the site is down, the crawler can give a copy from a public web archive. The crawler does not show the copy as a fresh fetch, and each result marks it.

const page = await crawler.fetch("https://example.com/hours", { fallback: "archive" });console.log(page.outcome, page.archived?.source, page.archived?.timestamp, page.doc);// blocked wayback 2024-01-15T09:30:00Z archive:https://example.com/hours

The fallback is off by default. The command line has fetch --archive-fallback, and the tool crawl_fetch has archive_fallback.

When the crawler gives a copy

Where the copy comes from

1. The Wayback Machine. Its availability API (archive.org/wayback/available) gives the newest copy, and the crawler fetches the raw copy (web.archive.org/web/<time>id_/<url>).
2. Common Crawl, when the Wayback Machine has no copy. The crawler reads the list of indexes (index.commoncrawl.org/collinfo.json) and asks the 3 newest indexes. It fetches the WARC record of the newest copy with a range request (data.commoncrawl.org).

• The archives are services of third parties, with their own politeness. Each request to them is a fetch of the same crawler. Their robots.txt, the pace of each host, the egress policy and the identity of the crawler apply.
• The capture log records each request to an archive, with the recipe archive-index (an index) or archive-copy (a copy). reindex reads the copies again from the log.
• A stored copy younger than maxAge answers with no request.
• An archive learns the URL of the page. A URL that holds a secret (a password or a secret parameter) does not go to an archive.

How a result shows a copy

• A copy is a document of its own, archive:<url>, of the kind archived. The document of the live page (url:<url>) does not change.
• fetch keeps the outcome and the status of the live fetch, for example blocked and 403. It adds archived: source (wayback or commoncrawl), timestamp (when the archive captured the page), url (the copy in the archive), original (the URL that the archive captured), and record for Common Crawl.
• read, verify and find give archived for a copy. The note of read and verify names the archive and the date. Thus a citation of a copy names the archive and its date, not the date of a fresh fetch.
• fetched_at stays the time of the fetch from the archive. archived.timestamp is the date of the copy.
• find with kind: "page" leaves the copies out.
• In the MCP envelope, a copy has the attribute archived, for example archived="wayback 2024-01-15T09:30:00Z". Its age starts at the date of the copy.
• With --archive-fallback, fetch gives the exit code 0 for a page with a copy, and the line of the page starts with ~.

robots.txt and the usage signals of a copy

An archive keeps copies of pages that the site refuses today. To read such a copy or not is a decision of policy, and the default is strict:

• With no archiveDespiteRobots, the crawler gives no copy of a page that the current robots.txt of its site disallows for this agent. The rules for the agents of onBehalfOf apply too.
• For a stopped host, the crawler reads the rules again: from memory, or from the stored copy of robots.txt of the last 30 days. Thus only a stop, an error or a dead host gets a copy.
• The reason: robots.txt tells what the owner of the site wants today. A copy of a disallowed page goes around that wish. An owner who disallows a page today does not expect an agent to read it from an archive.
• archiveDespiteRobots: true (--archive-despite-robots) gives a copy of these pages too. The robots.txt of each archive still applies to the requests to it. The MCP server has no such parameter, because this decision belongs to the operator.
• The usage signals of a copy apply as for a live page. A copy is held back from the AI reader when the content signals of the current robots.txt of its site say ai-input=no.
• The headers of the archived response count too: X-Archive-Orig-Content-Signal of the Wayback Machine, and the HTTP headers in the WARC record of Common Crawl. The signals of the archive itself count as well.

Signed requests (Web Bot Auth)

With an Ed25519 key, the crawler signs each request with HTTP Message Signatures (RFC 9421), as the IETF Web Bot Auth draft tells. A site can then verify that a request comes from the holder of your key. The package follows draft-ietf-webbotauth-httpsig-protocol-00 (1 September 2026).

import { readFileSync } from "node:fs";import { openCrawler } from "polite-crawler";
const crawler = openCrawler({  dir: "./crawl",  signer: {    privateKey: readFileSync("bot-key.pem", "utf8"), // PKCS#8 PEM, a JWK or a KeyObject    signatureAgent: "https://bot.example.com", // the origin of your key directory  },});

Each request gets three headers:

Signature-Agent: sig1="https://bot.example.com"Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1760702400;keyid="poqkLGiy...";alg="ed25519";expires=1760702700;nonce="...";tag="web-bot-auth"Signature: sig1=:...:

• The signature covers @authority and the member of Signature-Agent. keyid is the JWK thumbprint of the key (RFC 8037). Each signature has a new random nonce, and it expires after 5 minutes.
• The crawler signs each request: robots.txt, each redirect, the sitemaps, the archives of the fallback and each request of a rendered page. Without a signer, it signs nothing. The browser does not sign a WebSocket connection.
• A signature adds an identity. The User-Agent and From headers do not change. With a signer, the crawler replaces the signature headers of a caller.
• A signer that fails sends no request: the result is a skip with a note.
• The signature headers are not secrets. The capture log records signed, with the key id and the agent. It does not hold the key. The signer keeps the key in a closure, and no property of it shows the key.

createWebBotAuthSigner(options) makes a signer. signer also takes these options, or a RequestSigner of your own (for example, a signer with a key in a hardware module).

verifySignature(request, directory) verifies the signature of a request with a key directory, and tells why a request does not verify. Use it to test your setup.

Make a key with OpenSSL, and give it to the command line. Do not use the test keys of RFC 9421 for a real crawler.

openssl genpkey -algorithm ed25519 -out bot-key.pemnpx polite-crawler key-directory --sign-key bot-key.pem --signature-agent https://bot.example.com > http-message-signatures-directorynpx polite-crawler fetch https://example.com/ --sign-key bot-key.pem --signature-agent https://bot.example.com

The command line takes --sign-key and --signature-agent (or POLITE_CRAWLER_SIGN_KEY and POLITE_CRAWLER_SIGNATURE_AGENT) for fetch, discover, crawl and watch run, and polite-crawler-mcp takes them too. A --signature-agent with a path gives the type jwks_uri.

Host the key directory

A site finds your public key from Signature-Agent. With the type directory, the key directory is at /.well-known/http-message-signatures-directory on the origin of signatureAgent.

1. Make the JSON with keyDirectory([publicKey]) or polite-crawler key-directory. It holds only the public part of each key, with its thumbprint as kid.
2. Serve it at https://<your origin>/.well-known/http-message-signatures-directory. Send the status 200 and Content-Type: application/http-message-signatures-directory+json. Do not redirect it, because a verifier follows no redirect.
3. When you change the key, keep the old key in the directory until its last signature expires.

The draft recommends that the server of the directory also signs its response with each key, as a proof of possession. keyDirectory gives only the JSON, so a static host gives no such proof.

GitHub Pages cannot serve a key directory of the type directory for a project site:

• A project site (<user>.github.io/<repository>) serves files only below its path, so it cannot serve the well-known path at the root of the host.
• GitHub Pages sets the content type from the extension of the file, so it cannot send the type of the directory.
• On a project site, put the JSON in a file, for example jwks.json. Give its URL as signatureAgent, with agentType: "jwks_uri". The draft then binds the key to that URL, not to the domain.
• A host that sets headers (the _headers file of Cloudflare Pages or Netlify, for example) can serve the type directory at the root of a domain.

How the crawler is polite

1. robots.txt. The crawler reads the robots.txt of each site and obeys it, as RFC 9309 specifies. A 401 or a 403 for robots.txt means that the site has no file. When the crawler cannot read robots.txt (a dead domain, a server error or a body that does not decode), it fetches nothing from the site. Then it tries again after some minutes, with a backoff from Retry-After up to 1 hour. A good copy of the file from the last 30 days applies in that time.
2. One request at a time for each host. A host rests for 3 seconds or more between two requests, and longer after a slow response. Many hosts run in parallel, so a crawl is fast across sites and gentle on each one.
3. Stops. A 401 or a 403 stops the host for 1 hour, and a challenge or a 402 for 24 hours. Five failures in sequence stop the host for 1 hour. A Retry-After longer than 1 hour stops the host for that time, up to 30 days. The note of a skipped fetch says host stopped until <time>: <reason>. The crawler does not try again with a different identity.
4. Failures and backoff. A client error (a 404, for example), a 429 and a 503 do not count toward a stop. After an error or a request to wait, the crawler waits 5 seconds, then longer, up to 300 seconds, with random jitter.
5. An honest User-Agent. The default is PoliteCrawler/0.1.0 (+https://mark1russell7.github.io/crawler/bot/). The contact option adds an address. An email address also goes into the From header, and a URL does not, because From holds only a mailbox. The constructor of Crawler throws an error for a bad option. Examples are a token that is not a product token, a User-Agent or a From that HTTP cannot send, and a maxActive less than 1.
6. Public addresses only. By default, the crawler fetches only http: and https: URLs on public addresses. It checks the address after DNS and at each redirect. In the headless browser, it checks each request of the page and each redirect of that request. A refusal is a skipped fetch with the note egress policy: <address> is a loopback address; set allowPrivate to fetch it.
7. Usage signals. When a page asks not to be AI input (ai-input=no content signals), the crawler keeps it but holds back its text from the AI reader. The crawler also records the RSL License lines of robots.txt and the TDMRep reservation (the tdm-reservation and tdm-policy headers and <meta> tags) with each capture. These records change nothing, because a reservation of text and data mining is not a refusal of AI input.
8. Signed requests, as an option. With a signer, each request has a Web Bot Auth signature, so a site can verify where it comes from. Signed requests tells how.
9. Archived copies, as an option. fallback: "archive" reads a copy from a web archive only after a refusal. By default, it gives no copy of a page that the current robots.txt disallows.

To block the crawler, a site owner adds this group to robots.txt:

User-agent: PoliteCrawlerDisallow: /

Stored copies and conditional requests

A fetch uses the stored copy of a URL, with no request, while the copy is younger than maxAge (24 hours by default). With maxAge: "auto", the crawler uses the freshness that the server gave: Cache-Control, Expires, or 10 percent of the age of Last-Modified. With "auto", no-store and no-cache make a copy stale at once. A fixed maxAge ignores them. A response that varies on a request header (Vary) is used again only for a request with the same value of that header. The store keeps one copy for each URL, so a request with a different value replaces it.

When the stored copy is too old, the crawler sends a conditional request. If-None-Match sends the ETag of the copy, and If-Modified-Since sends its Last-Modified date. A 304 Not Modified sends no body, and the crawler uses the stored body again. The capture log records each request, with the validators that it sent (validators).

A server examines If-None-Match first, and it ignores If-Modified-Since when the request has the two (RFC 9110, section 13.1.3). Some hosts give a different ETag at each response, for example one for each edge server. Such a host does not answer 304 to the two validators, also when its Last-Modified date is stable. Thus the crawler keeps a validator memo for each host:

1. After 3 responses in sequence that send the stored body again, with If-None-Match and a Last-Modified date, the crawler stops the ETag for the host. It then sends only If-Modified-Since.
2. After 20 such requests, or after 7 days, the next request sends the ETag again (a probe). A 304 to the probe starts the ETag again. A 200 keeps it stopped until the next probe.
3. When the date alone also fails 3 times in sequence, the crawler sends the two validators again, because they cost nothing. The memo records that neither validator works.
4. A changed body tells nothing about a validator. The robots.txt of a host is not part of its memo, because it is often a static file with validators of its own.

The memo is in the host state store. openCrawler keeps it in hosts.sqlite, so the processes on one store share it, and it stays after a restart. MemoryHostStateStore keeps it in memory. A HostStateStore of your own gets a memo when it has the optional methods validators and noteValidators. nextValidatorMemo and VALIDATOR_MEMO give the rules and the limits.

The store

The package publishes the JSON Schema of a line of the log: polite-crawler/record.schema.json. A program that reads the log can check each record with it.

• The index keeps each capture of a document as a version. The current version is the latest capture by fetch time, or a render of the same page body. The earlier versions stay, so a citation that names a capture stays true after the page changes.

• The capture log and the bodies go to the disk before a fetch finishes. reindex skips a damaged line of the log and a damaged body, and it counts them.

• reindex builds a new index beside the old one and changes to it at the end. A rebuild that fails keeps the old index. The documents of a recipe that the runtime cannot read stay.

• At open, the crawler rebuilds an index of an earlier version. An index that is behind the log (a process stopped between the log and the index) gets the captures that it missed.

• An index of an earlier version of the package opens empty, and openCrawler builds it again from the log.

• Two or more processes can use one store. The index waits for the lock of a different process.

• A capture does not hold the secrets of a URL. The crawler writes REDACTED in place of a user name and a password. It does the same for the values of the usual secret query parameters (DEFAULT_SECRET_PARAMS) and of each name in secretParams.

• The crawler keeps no more than maxBytes of a body (5 MiB by default) after decompression. A longer body keeps its first part, and the capture says truncated. A body that does not download or decode gives the outcome error.

Export to a web archive

The store is private evidence, and it stays on your disk. An export is a decision of the operator: it copies the captures that you select into a standard web archive. Archive tools read it, for example warcio, pywb and ReplayWeb.page.

npx polite-crawler export --out crawl.wacznpx polite-crawler export --out site.warc.gz --host example.com --since 2026-10-01

import { exportWacz, exportWarc } from "polite-crawler";
await exportWarc("./crawl", { out: "crawl.warc.gz", host: "example.com" });
const result = await exportWacz(crawler.runtime, { out: "crawl.wacz", since: "2026-10-01" });console.log(result.responses, result.revisits, result.pages);

• exportWarc writes WARC/1.1 (ISO 28500:2017). Each record is one gzip member, so the file is a .warc.gz.
• exportWacz writes WACZ 1.1.1: a ZIP with the WARC file, a sorted CDXJ index, a page list and datapackage.json.
• The source is the folder of a store, or crawler.runtime. The export only reads the store. It makes no store, and it sends no request.
• These filters select the captures: since (this time or later), until (before this time), host (the host and its subdomains), urlPrefix and ids. The command line has --since, --until, --host, --prefix and --id.
• The export reads the capture log as a stream and holds one body at a time. It sorts the index and the page list in files on the disk. Thus its memory does not grow with the store.
• The export writes a temporary file and gives it its name at the end. A failed export leaves no part of a file. When the file exists, the export fails. Set overwrite (--overwrite) to replace it.
• The MCP server has no export tool. An export writes files on the disk of the operator, so it is an action of the operator, not of an assistant.

What goes into the archive

• The first record is a warcinfo record. It names the software, its version and the filters of the export.
• The metadata record of a capture (WARC-Concurrent-To) holds the JSON of the capture. It has the id, the kind, the URL, the times, the content hash and the payload. The payload has the outcome, the note, the robots.txt check and the usage signals. A reader can calculate the content hash again and check the capture id.
• Each record of a capture has the field WARC-Polite-Crawler-Capture with the capture id. The record ids come from the hash of the capture, so a second export gives the same ids.
• The digests are SHA-256 in base 32: sha256: and 56 characters, the form that warcio uses. WARC-Payload-Digest is the digest of the stored body.
• A revisit names its original (WARC-Refers-To and WARC-Refers-To-Date) when the original is in the same export. Its WARC-Payload-Digest is the digest of the body that the 304 confirmed.
• A body that the crawler cut at maxBytes has WARC-Truncated: length. A compressed body that ended early has WARC-Truncated: disconnect.
• The pages that ask not to be AI input go into the archive too. An archive is not AI input. Their usage signals are in the metadata record, so a reader of the archive can obey them.

What does not go into the archive, and why

• The raw headers. A capture keeps only some headers of the response: Content-Type, ETag, Last-Modified, Cache-Control, Expires, Age, Date and Vary. Thus the export rebuilds the status line and the header block from them, and it adds the Content-Length of the stored body. Such a record has the field WARC-Polite-Crawler-Headers: rebuilt, and its metadata record says "http_headers": "rebuilt".
  • The stored body is the decoded body. Thus the block has no Content-Encoding and no Transfer-Encoding.
  • The status line says HTTP/1.1, also when the server answered over HTTP/2.
  • The export writes a response record and not a resource record. Replay tools (pywb, ReplayWeb.page) replay a response with its status and its headers, and a resource record has neither. Browser crawlers also write the headers that they have, with a decoded body.
• The request. A capture does not keep the headers of the request (User-Agent and From). Thus the export writes no request record. The metadata record gives the validators that the request sent.
• The IP address. A capture does not keep the address of the server. Thus no record has WARC-IP-Address.
• The labels and the scope of the application. They can name a private lineage, so the metadata record leaves them out.
• The other records. The overrides of a held-back page and the records of the other sources of the log stay in the store.

The index and the pages of a WACZ file

• archive/data.warc.gz is stored with no more compression, so a replay tool reads a record at its offset.
• indexes/index.cdxj has one line for each response and revisit record, sorted by SURT key and time. The conversion records are not in the index, so a replay tool shows only the responses of the servers.
• pages/pages.jsonl has one page for each document: each URL with a 200 response, except robots.txt files. The page names its latest capture.
• datapackage.json gives the SHA-256 and the size of each part. datapackage-digest.json gives the SHA-256 of datapackage.json.
• The ZIP has Zip64 records only when a part or the file is larger than 4 GiB.

Contributing

CONTRIBUTING.md tells how to set up the repository and test a change. CHANGELOG.md records the changes in each version. Report a vulnerability privately, as SECURITY.md tells.

License

MIT