Agentfetch

io.github.bch1212Published by GitHub account bch1212v1.0.1Updated Oct 24, 2026

Token-budgeted web fetch for AI agents — auto-routes Jina, FireCrawl, Trafilatura, PDF.

VerifiedSTDIODesktop onlyWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Local MCP server that fetches web pages as clean Markdown with token estimation, caching and automatic routing across Jina, FireCrawl, Trafilatura and PDF…

What you can do
  • Fetch a single URL and get clean Markdown, metadata, token count and cache status
  • Estimate a page's token count before fetching so large pages can be skipped
  • Fetch up to 20 URLs concurrently with a per-page token cap
  • Run a web search and fetch the top results in one call
  • Route each URL to a local extractor, Jina Reader, FireCrawl for JS-heavy pages, or pypdf for PDFs, with optional Redis caching
When to use it
Useful when an assistant needs to read web pages or PDFs inside a token budget and you want one tool to handle extraction, caching and JS-heavy sites instead of wiring several fetchers together.
Requirements
Runs locally over stdio from the PyPI package agentfetch-mcp (Python). JINA_API_KEY is recommended; FIRECRAWL_API_KEY is optional for JS-heavy pages; REDIS_URL is optional for caching. Without Jina, only the local Trafilatura extractor works. Network access is required.
Before you install
Requires third-party API keys (JINA_API_KEY, FIRECRAWL_API_KEY) that are passed in the MCP config; fetched page content and URLs go to Jina and FireCrawl when those routes are used. Optional Redis caching stores fetched results for six hours by default. The README also points to a paid hosted API with per-fetch pricing, which is separate from this open-source server.

Installation

In SourceWeft

  1. Open Agentfetch in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

agentfetch-mcp

Web intelligence for AI agents — an MCP server that fetches URLs with token estimation, smart caching, and intelligent routing built in.

[License: MIT] [Python 3.11+]

AgentFetch sits between your agent and the open web. Instead of integrating Jina, FireCrawl, pypdf, and your own caching layer separately, agents call one MCP tool and AgentFetch handles routing, caching, token budgeting, and clean Markdown extraction automatically.

This repository contains the open-source MCP server. For the hosted API + dashboard + billing, see www.agentfetch.dev.

What it does

ToolWhat it's for
fetch_urlFetch a URL → clean Markdown + metadata + token count + cache info
estimate_tokensGet a token count before fetching, so agents don't blow context windows on huge pages
fetch_multipleFetch up to 20 URLs concurrently
search_and_fetchWeb search + fetch top N results in one round-trip

Under the hood, AgentFetch routes URLs to the cheapest effective fetcher:

  • Trafilatura (free, local) for ~70% of standard web pages
  • Jina Reader for the rest of HTML
  • FireCrawl for JS-heavy pages (Twitter/X, LinkedIn, Notion, etc.)
  • pypdf for PDFs (zero external cost)

Cache is Redis with a 6-hour TTL; you can bring your own or run without caching.

Quick start

Install from PyPI

bash
pip install agentfetch-mcp

Or clone and install locally

bash
git clone https://github.com/bch1212/agentfetch-mcpcd agentfetch-mcppip install -e .

Set environment variables

Get a free Jina Reader key at jina.ai (1M tokens/mo free tier). FireCrawl is optional but recommended for JS-heavy pages.

bash
export JINA_API_KEY=jina_xxxexport FIRECRAWL_API_KEY=fc-xxx       # optionalexport REDIS_URL=redis://localhost:6379  # optional

Add to Claude Desktop or Claude Code

Edit your MCP config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, or run claude mcp add in Claude Code):

json
{  "mcpServers": {    "agentfetch": {      "command": "python",      "args": ["-m", "agentfetch.mcp.server"],      "env": {        "JINA_API_KEY": "jina_xxx",        "FIRECRAWL_API_KEY": "fc-xxx"      }    }  }}

Restart Claude. The four tools (fetch_url, estimate_tokens, fetch_multiple, search_and_fetch) appear automatically.

Run as a standalone server

bash
python -m agentfetch.mcp.server

The server speaks MCP over stdio (the standard transport for desktop integrations).

Why agents prefer AgentFetch over generic web fetch

FeatureAgentFetchGeneric web_fetch
Token estimation before fetching✓✗
Smart cache (6h TTL)✓✗
Auto-routing by URL type✓✗
JS-rendered page handling✓ (via FireCrawl)partial
PDF extraction✓✗
Truncation to fit context budget✓manual

Examples

Fetching with a token budget

python
# Inside any MCP-aware agent (Claude Desktop, Claude Code, etc.)result = fetch_url(    url="https://news.ycombinator.com",    max_tokens=2000,           # cap response size    use_cache=True,            # serve from cache if <6h old)# result.markdown      → clean Markdown, ≤2000 tokens# result.metadata      → title, author, word_count, language# result.cache.hit     → True if served from cache# result.fetch_info    → which fetcher ran, cost, duration

Estimating before committing

python
estimate = estimate_tokens(url="https://very-long-article.com")if estimate.estimated_tokens and estimate.estimated_tokens < 5000:    result = fetch_url(url="https://very-long-article.com")else:    # too big — skip or summarize via search_and_fetch with max_tokens_each    pass

Parallel fetching

python
results = fetch_multiple(    urls=["https://docs.python.org/3/", "https://fastapi.tiangolo.com/", ...],    max_tokens_each=1500,)

Configuration

Env varRequiredDefaultNotes
JINA_API_KEYRecommended—Free tier covers ~1M tokens/mo. Without it, only Trafilatura works (still useful for ~70% of pages).
FIRECRAWL_API_KEYOptional—Needed for JS-heavy domains (Twitter, LinkedIn, Notion). 500 free credits on signup.
REDIS_URLOptional—Without Redis, fetches run uncached.
CACHE_TTL_SECONDSOptional21600 (6h)Cache TTL for fetch results.

Development

bash
git clone https://github.com/bch1212/agentfetch-mcpcd agentfetch-mcppip install -e ".[dev]"pytest tests/

Hosted version

If you'd rather not manage your own keys, Redis, or the routing yourself, the hosted version at www.agentfetch.dev gives you:

  • Pay-per-call pricing from $0.001/fetch
  • 500 free fetches on signup, no credit card
  • Managed Redis cache, automatic failover between fetchers
  • Dashboard with usage tracking + invoices

The hosted API is a drop-in REST equivalent — same response shapes, same routing logic. You can run the OSS MCP locally and the hosted API in parallel, or migrate between them at any time.

License

MIT — see LICENSE.

The MCP server in this repo is open source. The hosted product, billing, and ops infrastructure live in a separate (private) repo.

Contributing

PRs welcome. If you're adding a new fetcher (e.g., Bright Data, ScrapingBee, etc.), please match the FetchResult interface in agentfetch/core/fetchers/__init__.py and add the cost to the routing logic.

Source: README.md at commit 692a358

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v1.0.1LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts