Web Fetcher

mcp-server-web-fetcher

[CI] [npm version] [License: MIT] [TypeScript] [Node.js] [MCP] [MCP Registry]

A fast, dependency-light Model Context Protocol server that turns any web page into something a language model can actually read: clean Markdown, structured metadata, and a classified list of links.

Web pages are 90% chrome. Scripts, cookie banners, navigation, sidebars and tracking pixels burn context and derail reasoning. This server strips all of that on the way in, so the model sees the article and nothing else.

┌────────────┐   stdio/JSON-RPC   ┌──────────────────────┐   HTTPS   ┌──────────┐│ MCP client │ ─────────────────► │ mcp-server-web-      │ ────────► │ web page ││ (Claude,   │ ◄───────────────── │ fetcher              │ ◄──────── │          ││  Kiro, …)  │  Markdown + JSON   │ fetch → clean → md   │           └──────────┘└────────────┘                    └──────────────────────┘

Features

• Clean Markdown output. Readability-style main-content detection, GFM tables, language-tagged code fences, absolute links. No leftover HTML, even for gnarly nested tables.
• Context-window aware. Long pages are paginated with startIndex / nextStartIndex instead of being silently cut in half.
• Structured metadata. Title, description, canonical, lang, author, publish/modify dates, Open Graph, Twitter cards, JSON-LD, hreflang alternates, RSS/Atom feeds, h1–h6 outline and raw HTTP headers.
• Link intelligence. Absolute URLs, anchor text, rel, nofollow, internal vs external classification, scope filters, de-duplication.
• SSRF hardening. Loopback, private, link-local and cloud-metadata addresses are blocked on every redirect hop, not just the first request. Credentials in URLs are stripped.
• Predictable failures. Every error carries a stable machine-readable code (TIMEOUT, HTTP_ERROR, BLOCKED_HOST, RESPONSE_TOO_LARGE, …), a retryability flag and a recovery hint, so the model can self-correct instead of guessing.
• Typed end to end. Strict Zod input schemas plus published outputSchema, so clients get validated structuredContent, not prose they have to re-parse.
• Well behaved. Byte caps, timeouts, redirect limits, charset sniffing (including legacy encodings), a small TTL cache, transient-failure retries and optional robots.txt enforcement.
• 115 tests, no network required. Vitest unit tests plus in-memory MCP protocol tests.

Quickstart

Requires Node.js 20.18.1 or newer (inherited from cheerio, which needs undici 7).

# run it without installingnpx -y mcp-server-web-fetcher
# or install globallynpm install -g mcp-server-web-fetchermcp-server-web-fetcher

The server speaks MCP over stdio, so on its own it just waits for a client. Point a client at it:

Claude Desktop

~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows:

{  "mcpServers": {    "web-fetcher": {      "command": "npx",      "args": ["-y", "mcp-server-web-fetcher"]    }  }}

With configuration, using a global install:

{  "mcpServers": {    "web-fetcher": {      "command": "mcp-server-web-fetcher",      "env": {        "WEB_FETCHER_TIMEOUT_MS": "20000",        "WEB_FETCHER_RESPECT_ROBOTS": "true"      }    }  }}

Restart Claude Desktop, then ask it to summarise a URL.

Claude Code

claude mcp add web-fetcher -- npx -y mcp-server-web-fetcher

Kiro, Cursor, Windsurf and other MCP clients

Same shape, in .kiro/settings/mcp.json / the client's MCP config file:

{  "mcpServers": {    "web-fetcher": {      "command": "npx",      "args": ["-y", "mcp-server-web-fetcher"],      "disabled": false,      "autoApprove": ["fetch_page_markdown", "extract_metadata", "extract_links"]    }  }}

MCP Registry

The server is listed in the official MCP Registry as io.github.vojtisprime11/web-fetcher, so clients that read the registry can discover it directly:

curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.vojtisprime11/web-fetcher"

Docker

docker build -t mcp-server-web-fetcher .

{  "mcpServers": {    "web-fetcher": {      "command": "docker",      "args": ["run", "--rm", "-i", "mcp-server-web-fetcher"]    }  }}

The image runs as the unprivileged node user and carries production dependencies only. CI builds it and drives it with a real MCP client, so it is verified to start and answer introspection.

From source

git clone https://github.com/vojtisprime11/mcp-server-web-fetcher.gitcd mcp-server-web-fetchernpm installnpm run buildnode dist/index.js          # or: npm run inspect

{  "mcpServers": {    "web-fetcher": {      "command": "node",      "args": ["/absolute/path/to/mcp-server-web-fetcher/dist/index.js"]    }  }}

Tools

All three tools are read-only, idempotent and open-world (annotated as such in the protocol), and all take an absolute http(s) URL.

fetch_page_markdown

Downloads a page and returns clean, LLM-friendly Markdown.

Request:

{  "name": "fetch_page_markdown",  "arguments": {    "url": "https://example.com/blog/caching",    "maxLength": 8000,    "mainContentOnly": true,    "includeImages": false  }}

structuredContent:

{  "url": "https://example.com/blog/caching",  "requestedUrl": "https://example.com/blog/caching",  "status": 200,  "contentType": "text/html; charset=utf-8",  "title": "How caching works",  "markdown": "# How caching works\n\nCaching is the art of **not** doing work twice...",  "markdownLength": 7984,  "totalLength": 21874,  "startIndex": 0,  "endIndex": 7984,  "nextStartIndex": 7984,  "truncated": true,  "wordCount": 1203,  "bytesDownloaded": 148213,  "elapsedMs": 412,  "fromCache": false,  "redirects": [],  "metadata": {    "title": "How caching works",    "description": "A deep dive into HTTP caching.",    "canonical": "https://example.com/blog/caching",    "language": "en",    "author": "Ada Lovelace",    "publishedTime": "2026-01-15T09:00:00Z"  }}

To read the rest, call again with "startIndex": 7984. Keep going while nextStartIndex is not null.

extract_metadata

Everything a model needs to classify a page, without spending context on its body.

{  "name": "extract_metadata",  "arguments": { "url": "https://example.com/blog/caching", "includeJsonLd": true }}

structuredContent (abridged):

{  "url": "https://example.com/blog/caching",  "status": 200,  "charset": "utf-8",  "title": "How caching works",  "description": "A deep dive into HTTP caching.",  "canonical": "https://example.com/blog/caching",  "language": "en",  "author": "Ada Lovelace",  "publishedTime": "2026-01-15T09:00:00Z",  "modifiedTime": null,  "robots": "index, follow",  "favicon": "https://example.com/favicon.ico",  "openGraph": {    "title": "How caching works",    "type": "article",    "image": "https://example.com/img/cover.png"  },  "twitter": { "card": "summary_large_image", "site": "@example" },  "alternates": [{ "hreflang": "de", "href": "https://example.com/de/blog/caching" }],  "feeds": [    { "title": "Feed", "href": "https://example.com/feed.xml", "type": "application/rss+xml" }  ],  "jsonLd": [{ "@type": "Article", "headline": "How caching works" }],  "headings": [    { "level": 1, "text": "How caching works", "id": null },    { "level": 2, "text": "Directives", "id": "directives" }  ],  "httpHeaders": { "content-type": "text/html; charset=utf-8", "x-cache": "HIT" },  "wordCount": 1203,  "redirects": [],  "fromCache": false}

extract_links

{  "name": "extract_links",  "arguments": { "url": "https://example.com/blog/caching", "scope": "external", "limit": 50 }}

structuredContent:

{  "url": "https://example.com/blog/caching",  "status": 200,  "totalFound": 2,  "returned": 2,  "internalCount": 0,  "externalCount": 2,  "truncated": false,  "links": [    {      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching",      "text": "MDN article",      "title": null,      "rel": "noopener nofollow",      "internal": false,      "nofollow": true    }  ],  "fromCache": false}

www.example.com and example.com count as the same site; blog.example.com does not.

Error handling

Failures come back as MCP tool errors (isError: true), never as silent empty results:

HTTP_ERROR: 404 Client Error for https://example.com/missingRetryable: noHint: Check the URL, or retry later if the status is 429/5xx.

structuredContent.error carries the same information as JSON: { code, message, retryable, url, status }.

Timeouts, 5xx, 408 and 429 responses are retried once automatically before the error surfaces.

Configuration

Everything is optional; the defaults are safe.

Security

• SSRF guard on by default. Hostnames are resolved and checked against loopback, RFC 1918, CGNAT, link-local (including 169.254.169.254), multicast and IPv4-mapped IPv6 ranges — for the initial URL and every redirect target. localhost, *.localhost and *.internal are refused outright. Turning the guard off is an explicit opt-in.
• Byte and time caps. Responses are streamed and cut at the byte cap; every request is aborted at the timeout. Content-Length larger than the cap is rejected before download.
• No code execution. JavaScript is never run; <script>, <style>, <iframe> and friends are removed before conversion. Fetched content is data, not instructions — treat page text reaching a model as untrusted input.
• Credentials stripped. user:pass@ is removed from URLs before the request is made.
• Read-only. The server has no write, filesystem or shell surface, and no authentication state.

Found a vulnerability? See SECURITY.md.

Architecture

src/├── index.ts              # stdio entry point (logs to stderr only)├── server.ts             # transport-agnostic server factory + public API├── types.ts              # Zod input/output schemas for every tool├── tools/│   ├── fetchPageMarkdown.ts│   ├── extractMetadata.ts│   ├── extractLinks.ts│   ├── shared.ts         # result shaping, dependency injection│   └── index.ts└── lib/    ├── config.ts         # env-driven configuration    ├── errors.ts         # typed error codes + recovery hints    ├── net.ts            # URL parsing, SSRF guard, link resolution    ├── http.ts           # fetch pipeline: timeouts, caps, redirects, charset, cache    ├── robots.ts         # optional robots.txt support    ├── html.ts           # cheerio cleanup, metadata + link extraction    ├── markdown.ts       # turndown configuration, tables, pagination    └── cache.ts          # TTL + LRU cachetests/                    # unit tests + in-memory MCP protocol tests

Tool handlers take an injectable fetchImpl, so every test runs offline against a scripted HTTP stub. The library is also importable directly:

import { runFetchPageMarkdown } from 'mcp-server-web-fetcher';
const page = await runFetchPageMarkdown({ url: 'https://example.com' });console.log(page.markdown);

Development

The minimum supported Node.js version is 20.18.1, which comes from cheerio (it depends on undici 7, and undici 7 needs the File global that landed in Node 20). CI runs the suite on Node 20/22/24 and has a separate job that completes an MCP handshake against the built server on exactly 20.18.1, so the declared floor is verified rather than assumed.

npm installnpm run dev            # tsx watchnpm run typechecknpm run lintnpm test               # vitest runnpm run test:coverage  # thresholds enforcednpm run buildnpm run inspect        # MCP Inspector against dist/index.jsnode scripts/smoke.mjs https://example.com   # live end-to-end check

Contributions welcome — see CONTRIBUTING.md and CODE_OF_CONDUCT.md.

Limitations

• Static HTML only. Client-rendered SPAs return whatever the server sends; no headless browser.
• No PDF or Office document extraction.
• Sites behind bot protection (Cloudflare challenges, login walls) will return 403.
• Main-content detection is a text-density heuristic. Use mainContentOnly: false when a page has an unusual structure.

Roadmap

• search_page tool: return only the sections matching a query.
• Optional headless rendering behind a flag, for JS-only pages.
• Streamable HTTP transport in addition to stdio.
• Conditional requests (ETag / If-Modified-Since) in the cache layer.

Acknowledgements

Built on the MCP TypeScript SDK, cheerio, turndown and zod.

License

MIT © Vojta Holes and contributors