acrawl

io.github.Mingye-LuPublished by GitHub account Mingye-Luv0.17.0Updated Oct 30, 2026

Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.

VerifiedSTDIODesktop onlyWeb Search & ScrapingBrowser Automation

Overview

AI-generated overview

Autonomous web crawler and browser agent that lets an assistant navigate sites, extract data, and inspect network, console and performance.

What you can do
  • Navigate pages, click, type, select, hover and emulate phones, including inside your own logged-in browser via a bridge extension
  • Extract structured data from sites, switching from plain HTTP to a real browser for JavaScript-heavy pages
  • Inspect network requests, console logs, WebSocket frames, performance timings, cookies, storage and code coverage
  • Run axe-core accessibility audits and debug failing page flows
  • Run a goal-driven agent that plans pagination and returns JSON, and save crawls as deterministic scripts for replay without model calls
When to use it
Useful when you want an assistant to scrape or crawl sites end to end, debug a web page's network and console behavior, or repeat a crawl cheaply from cron or CI after exploring it once.
Requirements
Runs as a local process from a single binary (Linux, macOS or Windows). Browser features need Node.js 20+ and the bundled stealth browser. An LLM provider must be configured via the acrawl auth command, with your own API key; 25 providers are supported.
Before you install
It drives a real browser and can submit forms, click and type on live sites, so actions may change remote state. It uses your own LLM API key, which means model calls cost money. The bridge extension lets it act inside your logged-in browser sessions, so it can reach accounts you are signed into. Install scripts download and run release binaries.

Installation

In SourceWeft

  1. Open acrawl in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[acrawl]

Browser automation agent that acts and observes. Navigate, click, and extract — or inspect network traffic, debug console errors, profile performance, and audit accessibility. One Rust binary.

[CI] [License: MIT] [Rust] [Website]

Website · Docs · Install · Quick start · MCP server · Comparisons

Single Rust binary. Full DevTools observability. 43 tools. 25 LLM providers. MCP server built-in.

[acrawl architecture: the TUI, one-shot prompt, and MCP entry points feed a runtime turn loop that streams from an LLM provider and runs 43 tools, which fetch over HTTP, escalate to CloakBrowser, or drive the user's Chrome through a WebSocket extension bridge.]

text
acrawl > get the name, price and rating of every book on books.toscrape.com, all 50 pages

acrawl reads the goal, opens the site, works out the pagination and hands back JSON. Along the way the agent can watch the network, read the console, profile performance and audit accessibility, so it can debug a page as well as scrape it. It also runs as an MCP server, so Claude Code, Cursor, VS Code and 14 other clients can drive the same browser.

Install

Linux / macOS (x64, ARM64)

bash
curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash

Windows (x64, PowerShell)

powershell
irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex

The script downloads the latest release, verifies its SHA256 checksum and sets up CloakBrowser, the stealth browser acrawl uses for JavaScript-heavy pages. Browser features need Node.js 20+. To build from source, see the install guide.

Quick start

bash
acrawl auth anthropic          # or openai, google, other (Ollama, LM Studio, ...), 25 providers in allacrawl                         # interactive terminal UIacrawl prompt "list the titles of the top 10 stories on news.ycombinator.com"   # one goal, then exit

Then go further:

bash
acrawl mcp install                     # add acrawl to Claude Code, Cursor, VS Code and other MCP clientsacrawl run books_catalog -o books.json # replay a saved script with no LLM calls

The quick start walks through a first crawl, getting JSON out, and writing goals that work.

What you can do with it

Extract data from any site. Describe the fields you want; acrawl fetches static pages over HTTP and switches to a real browser only when a page needs JavaScript.

text
acrawl > get all repository names and star counts from github.com/trending

Debug a page like a developer would. Network requests, console logs, WebSocket frames, performance timings, cookies, storage, code coverage and axe-core accessibility audits are all agent tools.

text
acrawl > open localhost:3000/checkout, submit the form, and tell me which requests failed and why

Fill forms and work through flows. Click, type, select, hover, press keys and emulate phones. With the acrawl Bridge extension, it works inside your own logged-in Chrome, Edge or Brave.

Run it again for free. Save a crawl as a deterministic JSON script with loops, conditions and parallel branches, then replay it from cron or CI with acrawl run, with no model calls.

Split big jobs. The agent forks sub-agents onto separate tabs, and a URL-claiming registry stops two of them from crawling the same page.

Why acrawl

  • One binary. No Python environment, no Docker, no hosted service. Node.js is needed only for the bundled browser.
  • Acts and observes. Most browser agents navigate and click. acrawl's agent also sees the network, the console and performance, as first-class tools.
  • AI where you need it, scripts where you don't. Let the agent explore once; replay the result deterministically.
  • Your choice of model. 25 providers, including Anthropic, OpenAI, Gemini, DeepSeek, Bedrock, Azure, Vertex AI, GitHub Copilot, Groq, Mistral and OpenRouter, plus any OpenAI-compatible endpoint such as Ollama.
  • MCP both ways. Use other MCP servers as extra agent tools, or expose acrawl's 40 tools to your coding agent with acrawl mcp.
  • Open source. MIT-licensed, runs on your machine, with your own LLM key and no telemetry.

Bring your own model

[Anthropic, OpenAI, Google Gemini, DeepSeek, Amazon Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, GitLab Duo, Groq, Cerebras, Mistral AI, xAI, Cohere, Alibaba, OpenRouter, Together AI and Ollama]

Use it from your coding agent

acrawl mcp install sets acrawl up as an MCP server in any of these clients.

[Claude Code, Claude Desktop, Cursor, Windsurf, VS Code, OpenCode, Zed, TRAE, JetBrains, Gemini CLI, Qwen Code, Codex CLI, Hermes, OpenClaw, Goose, Crush and Aider]

How it compares

acrawlbrowser-usePlaywright MCPFirecrawl
What it isBrowser agent, CLI and MCP serverPython agent library and hosted agentMCP server for browser controlHosted scrape and crawl API
Own reasoning loopYes, localYes, local or hostedNo, your client decides each stepYes, hosted
InstallSingle Rust binaryPython packageNode packageHosted, or self-host
DevTools tools for the agentNetwork, console, WebSocket, performance, cookies, storage, coverage, axe-coreNot a focusConsole, networkNot a focus
Replays with no LLM callsYes, JSON scriptsNoNoNo
LicenseMITMITApache 2.0AGPL-3.0 engine

Reviewed October 2026. The full comparison has a page for each tool.

Documentation

Everything else lives on the website, mingye-lu.github.io/AgenticCrawler.

Get startedInstall · Quick start · LLM providers · MCP server
GuidesJavaScript-heavy sites · Deterministic scripts · Debugging with DevTools · Your own browser
ReferenceHow it works · All 43 tools · CLI and slash commands · Configuration · FAQ

Driving acrawl from a coding agent? npx skills add Mingye-Lu/AgenticCrawler installs the acrawl-mcp skill, which teaches the agent when to use direct tools, scripts or run_goal.

Development

acrawl is a Rust workspace of 11 crates.

bash
cargo build --releasecargo test --workspacecargo clippy --workspace --all-targets -- -D warningscargo fmt --check

See CONTRIBUTING.md to get started, CHANGELOG.md for release notes and SECURITY.md to report a vulnerability.

License

MIT

Source: README.md at commit e7f62f9

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

8
  1. v0.17.0LatestOct 29, 2026
  2. v0.16.0Oct 13, 2026
  3. v0.15.2Oct 12, 2026
  4. v0.15.0Oct 6, 2026
  5. v0.14.3Oct 1, 2026
  6. v0.14.2Sep 21, 2026
  7. v0.14.0Sep 20, 2026
  8. v0.13.5Sep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts