WebReaper

[logo]

WebReaper

[NuGet] [CI] [License: MIT]

AI-native web scraper. Single binary with a bundled Claude Code skill.

Built by HighCraft.io.

Install

macOS / Linux (Homebrew):

brew install alex-on-ai/webreaper/webreaper

Any POSIX shell (install.sh):

curl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh

.NET library:

dotnet add package WebReaper

Windows binaries are on the GitHub Releases page; winget and Scoop are on the v10.1 roadmap.

Updating: brew upgrade webreaper (Homebrew), or re-run the install.sh line with --upgrade appended (| sh -s -- --upgrade), or dotnet add package WebReaper (library). When a newer release exists, scrape / crawl / map print a one-line upgrade hint on stderr (interactive terminals only, never in a pipe or CI); disable that check with WEBREAPER_NO_UPDATE_CHECK=1.

30-second demo

$ webreaper scrape https://news.ycombinator.com# Hacker News
- [Show HN: ...](https://news.ycombinator.com/item?id=...)- [Ask HN: ...](https://news.ycombinator.com/item?id=...)...
$ webreaper initWrote WebReaper Agent Skill to .claude/skills/webreaper/SKILL.md
Try it out:  webreaper scrape https://example.com  webreaper map https://example.com

After webreaper init, the next Claude Code session picks up the skill and routes scraping intents ("give me the markdown of X", "what blog posts are on Y", "scrape the top 5 articles") to webreaper automatically.

Table of contents

• Why WebReaper
• Quick start
• Bot protection that just works
• Power your agent
• AI features
• Use cases
• Packages
• Compared to Firecrawl, Crawl4AI, and Crawlee
• API overview
• Architecture and interfaces
• Repository structure
• License

Why WebReaper

Quick start

CLI

# One page as Markdownwebreaper scrape https://example.com
# Save Markdown to a filewebreaper scrape https://example.com --output page.md
# Discover URLs on a sitewebreaper map https://example.com --search /blog/ --max-urls 50
# Crawl a whole site recursively (every on-domain page) to JSON Lineswebreaper crawl https://example.com > pages.jsonl
# Structured fields with a JSON schema (output: JSON; multi-page: JSON Lines)webreaper scrape https://example.com --schema schema.json
# Schema-free extraction with an LLM (bring your own OpenAI-compatible endpoint)webreaper scrape https://example.com --prompt "title and author" \  --model gpt-4o-mini --llm-url https://api.openai.com/v1
# Whole site, fields, cheaply: infer a schema once, then extract the restwebreaper crawl https://example.com --infer "product name and price" \  --model gpt-4o-mini --llm-url https://api.openai.com/v1 --output-dir ./out
# JS-rendered single-page appwebreaper scrape https://example.com --browser
# Bot-protected site: a plain scrape already auto-climbs HTTP -> browser on a# block; --stealth starts at a stealth backend (--auto-stealth = no prompt, for CI)webreaper scrape https://example.com --stealth
# Install the Claude Code skillwebreaper init

The CLI is built Native-AOT (ADR-0043), ships as a single binary on every tagged GitHub release across six RIDs (linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64, win-arm64), and is block-aware with automatic browser/stealth escalation (ADR-0083). The macOS binaries are Apple codesigned and notarized (ADR-0071); Homebrew installs run without Gatekeeper warnings on a clean machine.

Library

using WebReaper.Builders;
var engine = await ScraperEngineBuilder    .Crawl("https://news.ycombinator.com")    .AsMarkdown()    .WriteToConsole()    .BuildAsync();
await engine.RunAsync();

That is HTTP-only, no extra packages, no schema. For structured fields, swap AsMarkdown() for Extract(schema); for JS-rendered pages, Crawl for CrawlWithBrowser plus a transport satellite (WebReaper.Playwright or WebReaper.Cdp). The full surface is in the API overview.

Bot protection that just works

Most scrapers make you opt into a browser up front and guess when a site is blocking you. WebReaper detects the block and escalates on its own, one page at a time:

HTTP  ->  browser (Chromium)  ->  stealth (CloakBrowser)          (climbs only when a page actually looks blocked)

• Automatic. A plain scrape or crawl starts on a fast HTTP fetch and climbs to a real browser only when a page looks blocked (a challenge status, response header, or body marker). No flag needed for the browser fallback.
• Per page, host-sticky. The first confirmed block on a host lifts that host's floor, so the rest of a whole-site crawl starts at the working tier instead of re-paying the failed one. You pay for the climb once, not per page.
• No garbage in your data. A page still blocked at the top tier is dropped, never written to your output, and the run exits non-zero so an unattended job knows. Clean data or a clear signal, never a challenge page masquerading as content.
• You decide on stealth. --stealth starts at the stealth backend; --auto-stealth (or WEBREAPER_AUTO_STEALTH=1) enables it unattended; --no-auto-stealth caps the climb at a vanilla browser. The ~220 MB stealth backend downloads only if you opt in.

Self-hosted, single binary, no cloud round-trip (ADR-0083).

Power your agent

Onboarding an agent from zero? Hand it docs/AI-ONBOARDING.md — install → verify → choose-your-path on one page, no credentials anywhere.

Claude Code skill

webreaper init

Writes a polished SKILL.md to .claude/skills/webreaper/. Claude Code loads it on the next session and routes scraping intents to the CLI: "scrape the top 5 stories on HN and summarize each", "give me the markdown of this article", "this Cloudflare-protected site is blocking me". The skill describes when to prefer webreaper over the built-in WebFetch (artifacts and structured data go through webreaper; conversational answers stay on WebFetch).

MCP server

Two MCP servers expose the same tools (scrape, map, extract, extract_with_prompt, extract_inferred, crawl) for clients that speak the Model Context Protocol:

• WebReaper.Mcp (ADR-0049) speaks stdio, for local clients that spawn a process: Cursor, Claude Desktop, Copilot Studio.

• WebReaper.Mcp.AspNetCore (ADR-0086) speaks Streamable HTTP, for clients that connect to a URL. The headline case is n8n, whose MCP Client node is URL-only and cannot reach a stdio server. Run the Chromium-baked image, point n8n's MCP Client node at the URL with a bearer token, and call the tools from a workflow:

  bash

  docker run -p 8080:8080 -e WEBREAPER_MCP_TOKEN=your-secret \  ghcr.io/alex-on-ai/webreaper-mcp-http:latest

  See the n8n quickstart.

Both are thin facades over the library API; the primary agent surface remains the CLI.

CLI inside any agent harness

The single binary works inside any shell-spawning agent: LangChain ShellTool, OpenAI Assistants code-interpreter, GitHub Actions, internal scripts. Zero runtime to install; one syscall to invoke.

AI features

1. LLM-ready Markdown, no schema

await ScraperEngineBuilder    .Crawl("https://example.com")    .AsMarkdown()                   // ADR-0040 + ADR-0063    .WriteToConsole()    .BuildAsync()    .Result.RunAsync();

The Markdown extractor (HtmlToMarkdown primitive, ADR-0063; MarkdownContentExtractor adapter, ADR-0040) emits {title, markdown} per page. Pass the output straight into a follow-up LLM prompt.

2. Source-gen schemas with compile-time guards

using WebReaper.Extraction.Attributes;
[ScrapeSchema]public partial class Article{    [ScrapeField("h1")]                                              public string? Title { get; set; }    [ScrapeField(".views", Type = SchemaFieldType.Integer)]          public int Views { get; set; }    [ScrapeField(".tag", IsList = true)]                             public List<string> Tags { get; set; } = new();}
// Emitted at compile time, reflection-free, AOT-clean://   public static Schema Schema { get; }//   public static Article Materialize(JsonObject json)
var engine = await ScraperEngineBuilder    .Crawl("https://example.com/post")    .Extract(Article.Schema)    .Subscribe(p => HandleArticle(Article.Materialize(p.Data)))    .BuildAsync();

The WebReaper.Extraction.Generators Roslyn analyzer (ADR-0045) emits the schema and a Materialize function. Schema typos are compile errors; the generated path uses no reflection so it's AOT-clean.

3. LLM safety net for deterministic extraction

Three composable patterns where the LLM only fires when the deterministic path can't deliver. Same proposer-validator shape across all three.

using WebReaper.AI;
// (a) Fire LLM only when a field returns empty (ADR-0046).WithLlmFallback(chatClient)
// (b) Repair a broken selector once per (Schema, field), cache forever (ADR-0047).WithLlmSelfHealing(chatClient)
// (c) No schema at all: infer it from the URL, re-infer on validator failure (ADR-0067 + ADR-0069).UseAi(chatClient, AiPolicyMode.Inferred)

Stable pages cost zero LLM calls; broken pages cost one call per (page, field) and cache. Schema inference is one call per site (cached for the run). The WebReaper.AI satellite is built on Microsoft.Extensions.AI so any IChatClient works.

4. Autonomous agent: page selection by goal

using WebReaper.AI;
var result = await LlmAgent.RunAsync(    "https://example.com",    goal: "Find the contact email and phone number for the support team.",    chatClient);

AgentEngine (ADR-0051) runs a sequential decide → persist → execute loop over a closed-sum AgentDecision (Extract | Follow | Act | Stop). The brain picks each step from the bounded AgentState view; the engine validates (visited-link enforcement, MaxSteps cap), persists (IAgentRunStore), and dispatches. Durable resume across process restarts.

5. Semantic page actions

ScraperEngineBuilder    .CrawlWithBrowser(url, actions => actions        .Do(PageAction.SemanticAct("click 'sign in'"))   // ADR-0050        .Do(PageAction.WaitForNetworkIdle())        .Build())    .WithLlmActionResolver(chatClient)    // ...

A natural-language PageAction.SemanticAct(intent) is one of the ten closed-sum arms (ADR-0050 added it; ADR-0074 added Fill / Press / ScrollIntoView for form interactions). The transport resolves it once via the registered IActionResolver (LLM-backed by default), dispatches the concrete arm, and caches the resolution per crawl by intent string. First page pays the LLM, subsequent same-intent pages dispatch the cached arm with no LLM call.

Runnable end-to-end demo

Examples/WebReaper.AiNativeShowcase wires every feature in this section:

dotnet run --project Examples/WebReaper.AiNativeShowcase -- markdowndotnet run --project Examples/WebReaper.AiNativeShowcase -- sourcegendotnet run --project Examples/WebReaper.AiNativeShowcase -- llmdotnet run --project Examples/WebReaper.AiNativeShowcase -- routerdotnet run --project Examples/WebReaper.AiNativeShowcase -- changetrack

Use cases

• Build LLM context from blog or docs sites. webreaper map plus webreaper scrape per URL, piped into a prompt or a vector DB.
• Monitor competitor pricing or status pages for changes. Schedule the CLI with cron or a worker, store records in MongoDB or SQLite, plug in .WithChangeTracking() (ADR-0048) so the sink fires only on diff. Hash-based dedup; cron-friendly.
• Run an autonomous research agent. LlmAgent.RunAsync(url, goal, chatClient) decides which links to follow until the goal is met. Durable resume across restarts.
• Scrape Cloudflare-protected catalogs. A plain scrape auto-climbs HTTP to a browser on a block; add --stealth (or --auto-stealth for unattended runs) to escalate to a stealth backend. Blocked pages are dropped, never emitted as challenge-page garbage.
• Generate clean datasets from semi-structured pages. [ScrapeSchema] POCO plus the source generator; reflection-free, AOT-compiles into a native binary.
• Embed a scraping primitive in your own app. dotnet add package WebReaper; the public registration seam lets you plug Redis, Cosmos DB, your own sink.

Packages

The release ships fifteen packages (one core, fourteen satellites), all versioned in lockstep. The core stays dependency-light and Native-AOT-publishable with zero warnings; satellites bring their own SDK dependencies and quarantine them off the core graph (ADR-0009).

WebReaper.Cli (the AOT single-binary; ADR-0043) is not a NuGet package; it ships as platform binaries on every GitHub release (Native-AOT plus dotnet tool install are mutually incompatible on one target). Install via Homebrew or install.sh, or build from source.

Compared to Firecrawl, Crawl4AI, and Crawlee

Crawlee (Apify's Node/Python library) is also worth knowing; it covers similar ground to the WebReaper library API but doesn't ship a binary, a Claude Code skill, or a built-in LLM safety net. Use it if you're already in the Apify ecosystem.

The closest reference is Firecrawl: same AI-native positioning, opposite distribution shape. Firecrawl optimises for the hosted-API flow; WebReaper optimises for the local-binary flow. If you want a managed cloud with someone else's proxies and infra, Firecrawl is the buy. If you want a binary that runs locally with your own LLM key and no metering, WebReaper is the build.

API overview

The library is a fluent builder over a small set of seams. For the deep seam-by-seam reference (interfaces, main entities, custom sinks), see docs/architecture.md.

Schema extraction

using WebReaper.Builders;
var engine = await ScraperEngineBuilder    .Crawl("https://www.alexpavlov.dev/blog")    .Extract(new()    {        new("title", ".text-3xl.font-bold"),        new("text", ".max-w-max.prose.prose-dark")    })    .Follow("a.text-gray-900.transition")    .WriteToJsonFile("output.json")    .PageCrawlLimit(10)    .WithParallelismDegree(30)    .LogToConsole()    .BuildAsync();
await engine.RunAsync();

Each new("field", "css-selector") is a leaf; nest schemas for objects, set IsList = true for arrays, set Attr = "href" to read an HTML attribute instead of inner text.

Collect records in-process

Get the scraped records back in your own process, no file and no custom sink. Pass Subscribe (ADR-0038) the Add of a thread-safe collection, then read them after the run:

using System.Collections.Concurrent;using WebReaper.Builders;using WebReaper.Sinks.Models;
var records = new ConcurrentBag<ParsedData>();
var engine = await ScraperEngineBuilder    .Crawl("https://news.ycombinator.com")    .AsMarkdown()    .Subscribe(records.Add)            // called concurrently; collect into a thread-safe type    .BuildAsync();
await engine.RunAsync();
foreach (var record in records)    Console.WriteLine($"{record.Url}: {record.Data["markdown"]?.GetValue<string>()?.Length ?? 0} chars");

The Crawl driver fans out to sinks concurrently, so a ConcurrentBag is the right collector, not a bare List. For structured fields, read your schema keys out of record.Data instead of "markdown".

Parsing dynamic pages (SPA)

For JS-rendered pages, swap Crawl for CrawlWithBrowser and register a browser transport. Two satellites are available.

WebReaper.Playwright is the modern default (ADR-0053): multi-browser, all ten PageAction arms.

using WebReaper.Builders;using WebReaper.Playwright;
await ScraperEngineBuilder    .CrawlWithBrowser("https://example.com")    .Extract(new() { new("title", "h1") })    .WithPlaywrightPageLoader()    .BuildAsync();

WebReaper.Cdp is the AOT-clean alternative (ADR-0052): raw CDP over System.Net.WebSockets, AOT-publishable. Use for AOT consumers or as the base for stealth backends.

using WebReaper.Cdp;
.WithCdpPageLoader(new CdpLaunchOptions { Headless = true })// or .WithCdpPageLoader(cdpUrl: "http://localhost:9222") for BYO browser

For bot-protected sites, layer WebReaper.Stealth.CloakBrowser (ADR-0054) on top of WebReaper.Cdp:

using WebReaper.Stealth.CloakBrowser;
.WithCloakBrowser()    // auto-downloads CloakBrowser on first use, ~220 MB

For visible-browser debugging, add .HeadlessMode(false).

Running JavaScript and page actions

Drive the page as it loads. Pass an actions lambda.

using WebReaper.Builders;using WebReaper.Playwright;using WebReaper.Domain.PageActions;
await ScraperEngineBuilder    .CrawlWithBrowser("https://www.reddit.com/r/dotnet/", actions => actions        .ScrollToEnd()        .Build())    .Extract(new() { new("title", "h1") })    .WithPlaywrightPageLoader()    .BuildAsync();

PageActionBuilder exposes Click, Wait, ScrollToEnd, ScrollIntoView, WaitForSelector, WaitForNetworkIdle, EvaluateExpression, Fill, Press, SemanticAct, Repeat / RepeatWithDelay, and Build(). Fill(selector, value) / Press(key) / ScrollIntoView(selector) (ADR-0074) carry an implicit 30 s auto-wait and use the React-friendly native-setter trick on the CDP transport, so controlled components in React / Vue / Svelte observe the change. SemanticAct (ADR-0050) accepts a natural-language intent and resolves it via the registered IActionResolver (see AI features §5).

Persist progress locally

Survive kill -9 and resume across restarts. Two adapters: file-backed (zero deps, in core) and SQLite-backed (durable, opt-in via WebReaper.Sqlite).

using WebReaper.Builders;using WebReaper.Sqlite;
await ScraperEngineBuilder    .Crawl("https://example.com")    .Extract(new() { new("name", "h1") })    .Follow(".forumlink>a")    .Paginate("a.torTopic", ".pg")    .WriteToJsonFile("result.json")    .WithSqliteScheduler("crawl/state.db")        // resume is a query, not a position file    .TrackVisitedLinksInSqlite("crawl/state.db")  // the table is the set    .BuildAsync();

Pass dataCleanupOnStart: true to any sink, tracker, or scheduler method to wipe its store at start (note: WriteToJsonFile defaults this to true; the others default to false).

Authorization

If the site needs cookies, call SetCookies and fill the container. You perform the login yourself.

using System.Net;using WebReaper.Builders;
await ScraperEngineBuilder    .Crawl("https://example.com/protected")    .Extract(new() { new("name", "h1") })    .SetCookies(cookies =>    {        cookies.Add(new Cookie("AuthToken", "123"));    })    .BuildAsync();

Distributed and serverless

Swap the scheduler, config storage, and link tracker to Redis or Azure Service Bus; multiple workers or serverless functions share one crawl. Examples/WebReaper.AzureFuncs shows the serverless shape (two functions: StartScraping seeds the work, WebReaperSpider is the distributed Crawl driver). Examples/WebReaper.DistributedScraperWorkerService shows the worker-service shape.

DistributedSpiderBuilder.BuildSpider() returns a bare ISpider without a Crawl seed (ADR-0009 / ADR-0025: "two seams, not one bug" split). The worker's config is persisted separately by the start endpoint.

Storage and scheduler backends

Every backend is a swappable seam. In-memory is the default; file-backed lives in core; the rest come from satellites.

Custom sinks, the full interface index, and the main domain entities live in docs/architecture.md.

Repository structure

License

WebReaper is MIT-licensed (ADR-0017). All NuGet packages plus the WebReaper.Cli binary ship under the same terms. Use it commercially, embed it in proprietary software, fork it, modify it, redistribute it; the only ask is that you keep the copyright notice.

Prior to the 10.0.0 wave, WebReaper was GPL-3.0-or-later. The relicense is strictly more permissive: every existing user is unaffected; new users who couldn't embed under GPL now can. Historical contributors are credited in CONTRIBUTORS.md. See docs/adr/0017-relicense-gpl-mit.md for the analysis and contributor consent path.

Contributions are welcome under the same MIT terms; sign-off via DCO (CONTRIBUTING.md).

Built by HighCraft.io — a software and AI engineering team.