agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)

io.github.foomworksPublished by GitHub account foomworksv0.2.1Updated Oct 24, 2026

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

VerifiedStreamable HTTPWeb executableWeb Search & ScrapingKnowledge & Memory

Overview

AI-generated overview

Remote MCP server that fetches a public URL and returns clean, LLM-ready markdown, with robots.txt-respecting, read-only fetching.

What you can do
  • Fetch a single public URL and get back title, description, word count and full markdown
  • Preview a page cheaply by returning only the description and first ~600 characters
  • Request PNG screenshots or PDF renders of a page (paid, currently unavailable)
  • Check the render lane with a free discovery stub
When to use it
Useful when an assistant needs to read an article, documentation page or other static HTML page as clean text instead of raw HTML, and you want a reader that honors robots.txt and does not bypass paywalls or anti-bot measures.
Requirements
Remote streamable HTTP endpoint; no account, API key, headers or environment variables declared. Only publicly reachable URLs work; private, loopback and internal hosts are blocked. Static HTML only, with a body cap of about 2 MB and a fetch timeout of about 12 seconds.
Before you install
Screenshot and PDF tools are marked paid via x402 and currently return 503 until browser rendering is provisioned; the service states it holds no keys and never initiates payments. Fetched page content is sent to a third-party operator, and you are responsible for having the right to fetch the URLs you submit.

Installation

In SourceWeft

  1. Open agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free) in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.foomworks%2Fagent-web/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "agent-web-url-to-llm-ready-markdown-a-po": {
      "type": "http",
      "url": "https://agent-web.foomworks.workers.dev/mcp"
    }
  }
}

README

agent-web

URL → LLM-ready markdown. A charter-clean, polite, robots-respecting web reader for AI agents — honors robots.txt, identifies honestly, and never bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access.

🌐 Live: https://agent-web.foomworks.workers.dev · 🔌 MCP: https://agent-web.foomworks.workers.dev/mcp

What it gives an agent

Give it one publicly reachable URL; get back the page as clean, LLM-ready markdown — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM.

Connect as an MCP server (recommended)

Streamable HTTP, JSON-RPC 2.0:

https://agent-web.foomworks.workers.dev/mcp
ToolCostReturns
read_urlfreefetch a URL → full LLM-ready markdown (title, word count, markdown)
read_url_previewfreethe first ~600 characters of the markdown — a cheap look before the full pull

Or call it over HTTP

bash
BASE=https://agent-web.foomworks.workers.devcurl -s "$BASE/read?url=https://example.com/&ref=readme"          # full markdowncurl -s "$BASE/read/preview?url=https://example.com/&ref=readme"  # first ~600 chars

Discovery: /.well-known/mcp.json (MCP) · /.well-known/agent-card.json (A2A) · /openapi.json · /policy (Acceptable-Use Policy)

Charter-clean by design (the bright line)

This is a polite reader, not a scraper:

  • Honors the origin's robots.txt for our user-agent — a disallowed path is refused without fetching; an unconfirmable robots.txt (5xx/error) is treated conservatively as disallowed.
  • Identifies honestly with a descriptive user-agent — never spoofs a browser.
  • Read-only GET of the single URL you supply — no crawl, no proxy rotation.
  • Never bypasses anti-bot/CAPTCHA, paywalls, or login walls.
  • SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated.

You are responsible for your right to fetch the URL you submit.

Limits

  • Static HTML only for now — JavaScript-rendered pages, screenshots, and PDF are a later increment.
  • Body cap ~2 MB, fetch timeout ~12 s.
  • The service holds no keys and never initiates payments.

Install the AgentSkill

SKILL.md is a portable AgentSkill (ClawHub / skills.sh / Claude Skills).

About

agent-web is operated by FOOM — an AI-operated, human-supervised autonomous service. License: MIT.

Source: README.md at commit 24b8d20

Tools

5

read_url

Fetch one publicly reachable URL and return clean, LLM-ready markdown (title + page description + word count + markdown). HTML pages are extracted to markdown — including HTML tables, which become GitHub-flavored Markdown tables; URLs pointing straight at a Markdown or plain-text document (raw READMEs, llms.txt, docs) are passed through verbatim. Polite by design: honors the origin's robots.txt for our user-agent, identifies honestly, read-only GET, never bypasses anti-bot/CAPTCHA/paywalls. Free. JavaScript-rendered pages are not supported yet.

Parametersurlrequired

read_url_preview

Like read_url but returns only the page description + the first ~600 characters of the markdown (title + word count + a truncation flag) — a cheap way to check a page's relevance before pulling the full text. Free.

Parametersurlrequired

render_preview

Free discovery stub for the screenshot/PDF render lane. Same robots/SSRF guards as read_url. Currently returns a 503 (no charge) until Cloudflare Browser Rendering is provisioned on this account.

Parametersurlrequired

render_screenshot

PAID (x402): returns x402 payment instructions for a PNG screenshot of a publicly reachable URL, rendered via a real, robots-respecting headless browser. Use render_preview (free) first. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

ParametersurlrequiredwidthfullPage

render_pdf

PAID (x402): returns x402 payment instructions for a PDF render of a publicly reachable URL, via a real, robots-respecting headless browser. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

Parametersurlrequired

Version history

1
  1. v0.2.1LatestSep 16, 2026

Score

100% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts