WebCrawlerAPI

com.webcrawlerapiPublished by webcrawlerapi.comOfficial publisherv0.1.3Updated Nov 1, 2026

Turn any web page into clean Markdown for your AI assistant. OAuth sign-in, no API key.

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Remote MCP server that fetches any public web page and returns its main content as clean Markdown for an AI assistant.

What you can do
  • Fetch a public web page by URL and get its main content back as clean Markdown
  • Read JavaScript-rendered pages, with navigation, ads and boilerplate stripped
  • Pull docs, articles and product pages into a chat or knowledge base
  • Return a short human-readable error when a page is unreachable, blocked by robots.txt, times out, or the balance is empty
When to use it
Useful when an assistant needs to read specific web pages as text, such as documentation, articles or product pages, without a local crawler or API key setup.
Requirements
A remote Streamable HTTP endpoint with OAuth 2.1 sign-in; no API key to paste. A WebCrawlerAPI account and an approved organization are required, and the first request opens a browser sign-in. The only scope is markdown:read. Network access to the public web is needed.
Before you install
Each call is billed to the WebCrawlerAPI organization you approve, so usage costs money. The server receives the URLs you ask it to fetch and returns page content. Access is granted per organization and can be reviewed or revoked in the dashboard under Connected apps.

Installation

In SourceWeft

  1. Open WebCrawlerAPI in the dashboard and add it to a workspace.
  2. Connect the required credentials. They are stored privately and never shown on public pages.
  3. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/com.webcrawlerapi%2Fmcp-server/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config. The client completes OAuth sign-in on first connect.

{
  "mcpServers": {
    "webcrawlerapi": {
      "type": "http",
      "url": "https://mcp.webcrawlerapi.com/mcp"
    }
  }
}

README

WebCrawlerAPI MCP server

[CI] [License: MIT] [MCP Registry]

A remote Model Context Protocol server that lets AI assistants read any web page as clean Markdown. It runs on WebCrawlerAPI, so JavaScript-rendered pages work and navigation, ads and boilerplate are stripped. Use it to pull docs, articles and product pages into a knowledge base or into a chat.

  • Remote URL: https://mcp.webcrawlerapi.com/mcp
  • Transport: Streamable HTTP (stateless)
  • Sign-in: OAuth 2.1 through https://auth.webcrawlerapi.com, no API key to paste
  • Tool: get_markdown

You need a WebCrawlerAPI account. Sign up at https://dash.webcrawlerapi.com.

Connect

One-click install: Add to Cursor · Add to VS Code

In every client below, the first request opens a browser window where you sign in and approve access for one of your organizations. Requests are billed to that organization.

Claude Code

bash
claude mcp add --transport http webcrawlerapi https://mcp.webcrawlerapi.com/mcp

Then run /mcp inside Claude Code, pick webcrawlerapi and choose Authenticate.

Claude.ai and Claude Desktop

  1. Open Settings, then Connectors.
  2. Choose Add custom connector.
  3. Name: WebCrawlerAPI. URL: https://mcp.webcrawlerapi.com/mcp.
  4. Click Connect and approve access in the browser.

Cursor

Add this to ~/.cursor/mcp.json (or .cursor/mcp.json in a project):

json
{  "mcpServers": {    "webcrawlerapi": {      "url": "https://mcp.webcrawlerapi.com/mcp"    }  }}

Open Cursor Settings, then MCP, and click Connect next to webcrawlerapi.

VS Code

Add this to .vscode/mcp.json (or run "MCP: Add Server" and choose HTTP):

json
{  "servers": {    "webcrawlerapi": {      "type": "http",      "url": "https://mcp.webcrawlerapi.com/mcp"    }  }}

Start the server from the CodeLens above the entry and approve access in the browser.

MCP Inspector

bash
npx @modelcontextprotocol/inspector

Pick transport "Streamable HTTP", URL https://mcp.webcrawlerapi.com/mcp, then use the OAuth flow in the Auth tab and call get_markdown.

ChatGPT

  1. Open Settings, then Connectors, and turn on Developer mode (Advanced).
  2. Create a connector with URL https://mcp.webcrawlerapi.com/mcp and authentication set to OAuth.
  3. Enable the connector in a chat and approve access in the browser.

Menu names change between ChatGPT plans and releases. Custom connectors may not be available on every plan.

Tool reference

get_markdown

Fetch a public web page and return its main content as clean Markdown.

InputTypeRequiredDescription
urlstringyesAbsolute http:// or https:// URL, up to 2048 characters

Returns the page as Markdown text. On failure the tool returns an error result with a short human-readable message (for example an unreachable site, a page blocked by robots.txt, a timeout, or an empty balance).

Annotations: read-only, idempotent, not destructive, open world (it reaches out to the public web).

Each call is billed to your WebCrawlerAPI organization. See https://webcrawlerapi.com/pricing.

Authentication

The server is an OAuth resource server. Clients discover the authorization server from https://mcp.webcrawlerapi.com/.well-known/oauth-protected-resource, which points to https://auth.webcrawlerapi.com.

  • Scope: markdown:read (the only scope)
  • Tokens are bound to the MCP server URL (resource indicator) and to the organization you approve.
  • You can review and revoke connected apps in the dashboard under Connected apps.
  • The server never forwards your token to other services.

Privacy and terms

The server receives the URLs you ask it to fetch and returns the page content. It does not log tokens or full URLs with query strings.

Pricing

Usage is billed to your organization. Current prices: https://webcrawlerapi.com/pricing

Contributing

See CONTRIBUTING.md and the Code of Conduct. Changes are listed in CHANGELOG.md.

Support

[email protected]. Security reports: see SECURITY.md.

Development

Requires Node 22 and pnpm.

bash
pnpm installcp .env.example .env   # fill in the valuespnpm dev               # tsx watchpnpm lint && pnpm test && pnpm build

Configuration is via environment variables, see .env.example and CONTRIBUTING.md. Registry publishing steps are in docs/registry-verification.md.

License

MIT, see LICENSE.

Source: README.md at commit c48b9c5

Tools

0
This server asks you to sign in, so SourceWeft could not read its tools without an account and has not indexed them. What a connected account sees can depend on that account and its permissions.

Version history

1
  1. v0.1.3LatestNov 1, 2026

Score

85% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts