Robotbase Content

Search the live web and get ranked results with title, URL, snippet and published date. When to use: an agent needs current facts, news or links that are not on chain - 'what is the BTC fee market saying today', 'recent Zcash governance proposals', 'find the docs page for X'. Use it before web_fetch, then fetch the one URL that looks authoritative. Do not use: for on-chain facts (use the BTC/ETH/XMR/ZEC/TRON tools) or for content inside a CyberLibrary book (use cyberlibrary_fulltext_search). Provider: self-hosted SearXNG (Bing/Baidu/360/Sogou/Yandex/Marginalia) first, then Serper, Tavily and cn.bing RSS as automatic fallbacks. Returns: {query, count, results:[{title,url,snippet,published}], provider (serper|tavily|bing_rss), cached, took_ms, _metering}. Results may be cached for up to 10 minutes; a cached result still counts as one call against the free allowance. Metered: all web tools share 100 free calls/day per IP (separate from the risk/library allowance), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[]. max count is 10.

Fetch one public web page and return its readable text as clean markdown or plain text, with navigation, scripts, headers and footers stripped. When to use: you have a specific URL (from web_search or the user) and need what the page actually says - 'read this article', 'summarise this docs page', 'turn this URL into markdown'. Do not use: to crawl or mirror a site, to fetch a page behind a login or paywall, to read API/RPC endpoints, or for book text inside CyberLibrary (use cyberlibrary_get_chapter). Returns: {url, final_url, status, title, content, chars, truncated, fetched_at, source (direct|provider), cached, took_ms, _metering}. The response is capped at 12000 characters per call; raise max_chars (up to 20000) when you need more. Reads one public URL only, respects robots.txt, and does not bypass login or paywall. Direct fetch is tried first; pages blocked from this node are fetched through a provider and marked source=provider. Metered: all web tools share 100 free calls/day per IP (separate from the risk/library allowance), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[].

Capture a screenshot of one public web page as an image (JPEG) plus metadata. When to use: visual layout matters - a dashboard, chart, UI state, or 'what does this page look like'. Do not use: to read text (use web_fetch), for pages behind a login or paywall, or for bulk capture. Returns: image content (image/jpeg) plus {url, final_url, title, width, height, bytes, engine, elapsed_ms, _metering}. Rendered by a real Chromium browser, robots.txt-respecting, SSRF-guarded, max 3MB per image. Metered: shares the web free bucket (100 calls/day per IP shared by all twelve web tools), then 2 credits per call over x402 - there is no API key; the 402 answer carries accepts[].

Discover the URLs of one site - sitemap first, then same-host links from the homepage. When to use: you need the list of pages a site exposes before fetching a few of them. Do not use: as a crawler (max 50 URLs, no page content) or to bypass robots.txt. Returns: {url, final_url, count, urls:[{url,source}], cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[].

Fetch up to 5 public URLs in one call and return each page's title, engine and a short snippet (first ~1200 characters), so an agent can triage several links quickly. When to use: search returned several candidate pages and you want a first pass over all of them; then call web_fetch on the ones worth reading in full. Do not use: when you need the complete text of a page (use web_fetch), or to crawl a site (use web_map). Returns: {ok, count, ok_count, results:[{url, ok, final_url, status, title, engine, chars, snippet, error?}], took_ms, _metering}. Partial failures do not fail the whole call. Metered: counts one call per URL against the shared web free bucket (50 URLs/day per IP together with web_fetch), then 1 credit per URL over x402 - max 5 URLs per call, no API key.

Read an RSS or Atom feed and return its latest items. When to use: the user wants the latest posts/news from a site that publishes a feed, or you found a feed URL and want headlines plus links. Do not use: to read article text (use web_fetch) or to discover a site's URLs (use web_map). Returns: {ok, url, final_url, title, count, items:[{title, link, published, summary}], cached, took_ms, _metering}. Summary is HTML-stripped and capped at 500 characters. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key.

Extract a page's structured metadata: Open Graph, Twitter cards, canonical URL, language and JSON-LD. When to use: you need the title/description/author/image of a page, or JSON-LD structured data for a product/article, without reading the whole page. Do not use: when you need the page text (use web_fetch) or a screenshot (use web_screenshot). Returns: {ok, url, final_url, title, description, image, site_name, type, canonical, lang, jsonld[], cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key.

Crawl one site's same-host pages starting from a URL (depth 1-2, at most 20 pages) and return each page's title, URL and readable text. When to use: you need several pages from one site (docs section, news index, product list) and web_map alone is not enough. Do not use: to mirror a whole site (hard cap 20 pages, same host), or for one page (web_fetch). Returns: {ok, url, final_url, count, pages:[{url, title, status, chars, content}], cached, took_ms, _metering}. Respects robots.txt and SSRF guards; partial page failures are skipped. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 3 credits per call over x402 - no API key.

Extract structured data from one public page: tables (headers + rows), JSON-LD blocks, and Open Graph/Twitter/canonical/language metadata. When to use: the page has tables or schema.org JSON-LD you need as data, without an LLM. Do not use: when you only need the page text (web_fetch) or a screenshot (web_screenshot). Returns: {ok, url, final_url, title, meta:{description,image,site_name,type,canonical,lang}, jsonld[], tables:[{caption,headers,rows}], links? , cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402.

Drive a real Chromium browser on one public page: click, type, press keys, scroll, wait for selectors, then return the resulting text (and optionally a screenshot and JSON XHR responses). When to use: the page needs interaction before its content appears (cookie banners, tabs, 'load more', form submit) and web_fetch/web_crawl cannot see it. Do not use: for plain reading (web_fetch) or bulk crawling (web_crawl); no login/paywall bypass. Returns: {ok, url, final_url, title, markdown, chars, elapsed_ms, screenshot?, xhr?, took_ms, _metering}. Max 10 actions; screenshot returns MCP image content. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 3 credits per call over x402.

Submit a same-host crawl as an asynchronous job and return a job_id immediately. When to use: you need more than a couple of pages and cannot hold a long HTTP request; submit here, then poll web_crawl_job until status is done, failed or cancelled. Do not use: for one page (web_fetch) or when you need the result in this call (web_crawl). Returns: {ok, job_id, status, url, depth, max_pages, max_chars, idempotent, poll_with}. Pass idempotency_key to retry safely (24h window returns the same job). Metered: shares the web free bucket (100 calls/day per IP), then 3 credits per submit over x402.

Check or cancel an asynchronous crawl job created by web_crawl_async. When to use: after web_crawl_async, poll with action=status until status is done/failed/cancelled; use action=cancel to stop a running job. Do not use: to start a crawl (web_crawl_async) or for a quick one-page read (web_fetch). Returns status: {ok, job_id, status: queued|running|done|failed|cancelled, progress:{pages_done,max_pages}, created_at, updated_at, result?, error?}. Free (not metered); rate-limited. Done result uses the same shape as web_crawl.

Extract a page's structured data into a named schema's common fields (no LLM): article, product, event, faq, recipe, course, organization, local_business, breadcrumb, dataset or job_posting. When to use: you need fields (title/author/date, price/currency/availability, event dates, FAQ Q&A, recipe ingredients, org address) rather than raw markdown or tables. Do not use: to get every JSON-LD block verbatim (web_metadata/web_extract) or to read text (web_fetch). Returns: {ok, url, final_url, schema, requested_schema, detected_types, matched, fields:{...}, sources:[jsonld|meta], jsonld_count, tables_count, took_ms, cached, _metering}. If no matching JSON-LD exists it falls back to Open Graph/meta fields and matched=false. Metered: shares the web free bucket (100 calls/day per IP), then 1 credit per call over x402.

Fetch a page now and compare it with the previous snapshot of the same URL, returning whether it changed and a unified diff (no LLM). When to use: monitoring a page for updates - prices, docs, policy pages, SEO titles/meta; first call stores the baseline, later calls return added/removed lines. Do not use: for one-off reading (web_fetch) or structured fields (web_extract_schema). Returns: {ok, url, baseline, changed, diff, added_lines, removed_lines, previous_ts, current_ts, title, hash, chars, fetched_cached, took_ms, _metering}. fresh=true (default) adds a cache-busting query to bypass the 6h page cache; pass fresh=false to compare cached content. Snapshots are kept per URL in SQLite (latest only). Metered: shares the web free bucket (100 calls/day per IP), then 1 credit per call over x402.

Create a scheduled change monitor for one public URL: the server re-fetches the page on an interval, stores a new snapshot and records a diff event whenever the readable text changes. When to use: the user wants to be told when a page changes (price, docs, policy page, job post, release notes) and cannot poll it themselves; also for handing off 'watch this for me'. Do not use: for a single comparison right now (web_diff) or when you only need the page today (web_fetch). Returns: {ok, monitor_id, url, status:active, interval_minutes, next_check_ts, webhook_enabled, webhook_secret?, webhook_signature?, hint}. The first check runs within ~60s to store the baseline; later changes show up in web_monitor_list.recent_events and, if requested, are POSTed to webhook_url. Webhook: https only, no redirects, 8s timeout, up to 2 retries (30s/120s); body JSON {monitor_id,url,ts,changed,added_lines,removed_lines,title,diff,event_id}; verify header 'X-RobotBase-Signature: sha256=<hex HMAC-SHA256 of the raw body using webhook_secret>'. Limits: 15-1440 minutes; max 5 monitors per IP; monitors expire after 7 days. Metered: 1 credit (shares the web free bucket of 100 calls/day per IP).

List the scheduled change monitors owned by the calling IP, with latest/next check time, last title or error, and the 5 most recent change events. When to use: after web_monitor_set, poll this instead of re-fetching the page; or to show the user what is currently being watched. Do not use: to create a monitor (web_monitor_set) or to compare one page now (web_diff). Returns: {ok, count, monitors:[{monitor_id,url,interval_minutes,active,created_ts,last_check_ts,next_check_ts,last_title,last_error,webhook_enabled,events_total,recent_events}], note}. Pass monitor_id to get that one monitor plus last_event.diff (the most recent detected change, up to 4000 chars) so you can read what changed without a webhook. Free (not metered); rate-limited to 20 calls/min per IP.

Delete one scheduled change monitor owned by the calling IP (its event history is removed too). When to use: the user no longer needs to watch a page, or the per-IP monitor limit was reached. Do not use: to pause a monitor temporarily - there is no pause; delete and re-create. Returns: {ok, deleted, monitor_id, note}; unknown or foreign ids return {ok:false,error:'monitor_not_found'} and delete nothing. Free (not metered); rate-limited to 20 calls/min per IP.

Cross-platform social hot lists: Baidu, Toutiao, Douyin, Bilibili and Hacker News. When to use: what people are talking about now, content ideas, trend discovery. Do not use: for search interest over time (attention_trend) or general web search (web_search). Returns: {platform, count, items:[{title, heat, url, source}], errors}.

Attention and search-interest signals: Google Trends RSS, Wikipedia pageviews and Google Books Ngram. When to use: is a topic rising, long-term attention, regional trending searches, evidence of public interest. Do not use: for social hot lists (social_hot) or prediction-market odds (prediction_market_odds). Returns: {query, geo, trending[], pageviews, ngram, errors}.

Metadata for exactly one book: title, author, genre, chapter count, word count, source URL and every access URL (HTML page, Markdown mirror, first chapter). When to use: you have a lang+slug and need to decide whether the book is worth fetching, or you need chapter_count before pulling a specific chapter. Do not use: to read content (call cyberlibrary_get_chapter or cyberlibrary_get_markdown). Returns: one catalog record object.

Full-text search inside the actual book text (a 26 GB corpus of chapter Markdown), returning the matching passage plus the book, chapter and canonical source URL for each hit. When to use: you need to locate where a phrase, name or quotation occurs in the corpus, or to ground an answer in a verifiable passage. Do not use: to find a book by title (call cyberlibrary_search_books, which is far cheaper). Strongly prefer passing lang: English alone is 79% of the corpus (20.4 GB), so a language-scoped search is typically 10-100x faster and more relevant. Returns: {query, lang, searched_languages, skipped_languages, scanned_mb, elapsed_ms, results[{lang, book, chapter, source_url, match, snippet}]}.

Search the catalog by title or author within one language, returning matching book records (title, author, lang, slug, chapter_count, word_count and URLs). When to use: you have a title or author and need the lang+slug that every other tool requires. Do not use: to find words inside the text of books - that is cyberlibrary_fulltext_search. Returns: {query, lang, searched_languages, matches_returned, books[']}.

Ranked passage retrieval over the paragraph index (BM25). Use a natural-language or keyword query to find the most relevant passages across books, with the book, chapter, chunk_id, citation, license and source URL for each hit. When to use: you need evidence passages for a question rather than a literal phrase match. Do not use: to verify an exact quotation (cyberlibrary_verify_quote) or to read a whole book (cyberlibrary_get_chunks). Pass lang when known (faster/more precise). Returns: {query, matched, candidates, corpus_version, results[{chunk_id, score, lang, slug, title, chapter, para, citation, url, license, snippet}]}. Note: BM25 lexical ranking; a neural semantic layer is planned.

Semantic passage search over a neural embedding index: describe what you are looking for in natural language and get the closest passages, each with a permanent chunk_id, book/chapter/paragraph, citation, license and source URL. When to use: keyword/BM25 search misses paraphrases or themes ("a story about a soldier returning from war", "清代公案小说里的侠客"). Do not use: to verify an exact quotation (cyberlibrary_verify_quote) or to read a whole book (cyberlibrary_get_chunks). Falls back to BM25 if the embedding service is down. Returns: {query, lang, backend, total_vectors, corpus_version, results[{chunk_id, score, title, author, chapter, para, citation, url, license, snippet}]}.

Retrieve ranked passages for a question and pack them into a token budget, ready to paste into an agent context. Each passage carries a permanent chunk_id, book/chapter/paragraph, citation, license and source URL, plus a packed context string and citations index. When to use: you are building a RAG answer and need grounded evidence with citations, without assembling chunks yourself. Do not use: to verify an exact quotation (cyberlibrary_verify_quote) or to read a whole book (cyberlibrary_get_chunks). Returns: {query, lang, budget_tokens, used_tokens, passages_returned, corpus_version, context, passages[], citations[]}.

Clean Markdown of a single chapter - the recommended way to put Cyber Library text into an agent context. Chapter 0/1 are both accepted for the opening chapter. When to use: after cyberlibrary_get_toc, to read the specific chapter you care about without pulling the whole book. Do not use: to read an entire book in one shot (use cyberlibrary_get_markdown, but mind the size). Returns: {chapter, title, chars_total, chars_returned, truncated, markdown, markdown_url}.

Whole-book Markdown mirror, including the book's own metadata header and summary. When to use: you need the complete text and have checked chapter_count/word_count via cyberlibrary_book_meta, or you specifically want the book-level summary block. Do not use: for long books where you only need part of the text - prefer cyberlibrary_get_toc + cyberlibrary_get_chapter, which is far cheaper. Returns: {chars_total, chars_returned, truncated, markdown, markdown_url}.

Every language catalog in Cyber Library with item counts and the corpus generation date. When to use: first call in any session - it tells you which languages exist and how large each is, so you can pick a language before fetching a shard. Do not use: to list actual books (call cyberlibrary_language_shard) or to read a book (call cyberlibrary_get_markdown / cyberlibrary_get_chapter). Returns: {generated_at, total_items, readable_items, languages[]} where each language entry carries {lang, total_items, url|pages}.

Takes no parameters.

Table of contents for one book: ordered chapters with their real titles. When to use: right after cyberlibrary_book_meta, to pick which chapter to read; this is the cheapest way to survey a long book (single HTML read regardless of chapter count). Do not use: to read prose (call cyberlibrary_get_chapter). Returns: {lang, slug, total_chapters, page, total_pages, chapters[{chapter, title, url}]}.

Ready-to-paste citation for one book: title, author, canonical page URL, Markdown URL, license and the public-domain territory notice. When to use: before quoting or summarising a book, so the answer carries a verifiable source. Do not use: to fetch content (use cyberlibrary_get_chapter). Returns: {title, author, page_url, markdown_url, license, source_url, citation, notice}.

Verify whether a quotation actually occurs in the Cyber Library corpus and return the exact passage, book, chapter, permanent chunk_id and citation. When to use: before citing a quotation, to catch model hallucination, or to ground a claim in a verifiable source. Do not use: to find a book by title (cyberlibrary_search_books) or to read a chapter (cyberlibrary_get_chapter). Matching is exact substring first, then fuzzy (>=90% of the quote). Returns: {quote, verified, match_type, best_score, matches[{chunk_id, score, title, chapter, para, citation, url, license}], corpus_version}.

Stable paragraph-level chunks for one book, each with a permanent chunk_id (lang:slug:chapter:para), citation text and canonical URL. When to use: you need RAG-ready passages with deterministic IDs instead of whole chapters, or you want to cache/index exact passages. Do not use: to read a whole chapter (cyberlibrary_get_chapter) or to search across books (cyberlibrary_verify_quote / cyberlibrary_fulltext_search). Returns: {lang, slug, title, license, corpus_version, total_chunks, chunks[{chunk_id, chapter, para, text, citation, url}]}.

One page of the book catalog for a language. Large languages are paginated (for example en = 25 pages x 2500 books); small languages are a single shard. When to use: browse or sample what a language holds, or locate a book title. Do not use: to search by keyword across the whole corpus (that is the planned cyberlibrary_fulltext_search) or to get one book's metadata when you already know its slug (call cyberlibrary_book_meta). Returns: {lang, page, total_pages, total_items, count, books[]} with books trimmed to `limit`.

Discovery endpoints for Cyber Library: MCP endpoint, tools list, Server Card, usage stats, catalog API, llms.txt / llms-full manifest, OpenAPI and the HTML search UI. When to use: you need a machine-readable entry point or want to know what else this service exposes. Do not use: for book data itself. Returns: a flat map of named endpoint URLs plus version and citation guidance.

Takes no parameters.

Unified CyberLibrary search: one query, automatic language, BM25 + neural semantic fusion (RRF), returning the most relevant passages with stable chunk IDs, citations, licenses and source URLs, plus an answerable/confidence signal. When to use: first choice when you do not want to decide between keyword and semantic search. Modes: auto (default, both), keyword (exact terms/names), semantic (paraphrase/theme). Options: lang, limit, snippet_chars. Returns: {query, lang, mode_requested, mode_used, answerable, confidence, corpus_version, results[{chunk_id, score, sources, title, author, chapter, para, citation, url, license, snippet}], next_actions}.