agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)

io.github.foomworks由 GitHub 账号 foomworks 发布v0.2.1更新于 Oct 24, 2026

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key

已验证Streamable HTTP可网页运行网页搜索与抓取知识与记忆

概览

AI 生成的概览

远程 MCP 服务器，抓取公开网址并返回干净的、适合大模型使用的 Markdown，遵守 robots.txt 且只读。

可以做什么
  • 抓取单个公开网址，返回标题、描述、字数与完整 Markdown
  • 低成本预览页面，仅返回描述与前约 600 个字符
  • 请求网页的 PNG 截图或 PDF 渲染（付费，目前不可用）
  • 用免费探测工具检查渲染通道状态
适用场景
当助手需要把文章、文档页等静态 HTML 页面读成干净文本而非原始 HTML，并且希望读取方遵守 robots.txt、不绕过付费墙或反爬机制时适用。
运行要求
远程 streamable HTTP 端点；未声明需要账号、API 密钥、请求头或环境变量。仅支持公开可访问的网址，私有、回环与内网主机会被拦截。仅支持静态 HTML，正文上限约 2 MB，抓取超时约 12 秒。
安装前请注意
截图与 PDF 工具标注为通过 x402 付费，且在浏览器渲染开通前返回 503；服务方称自身不持有密钥、不会发起支付。抓取的页面内容会发送给第三方运营方，且你需要对所提交网址拥有抓取权利。

安装

在 SourceWeft 中

  1. 打开 控制台中的 agent-web — URL to LLM-ready markdown: a polite, robots-respecting web page reader (free)，将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Web executable，通过 Streamable HTTP。 远程服务在工作区中配置后即可从网页运行时运行。

交给智能体安装

把这段提示词粘贴到 Claude Code、Codex、Cursor 等智能体，或 SourceWeft 对话中。智能体会读取该服务器的安装指南，先向你展示发布者、端点和登录方式，经你同意后再把它添加到你的客户端。密钥不会写进任何命令。

阅读 https://sourceweft.com/mcp/io.github.foomworks%2Fagent-web/install.md，并按其中说明把这个 MCP 服务器接入我的客户端。修改任何配置之前，先告诉我它的发布者、端点和登录方式，等我确认。不要把 API 密钥写进命令或我会提交的文件；问我密钥应从哪里获取。

查看智能体所依据的安装指南

其他 MCP 客户端

把它添加到你客户端的 mcpServers 配置中。

{
  "mcpServers": {
    "agent-web-url-to-llm-ready-markdown-a-po": {
      "type": "http",
      "url": "https://agent-web.foomworks.workers.dev/mcp"
    }
  }
}

README

agent-web

URL → LLM-ready markdown. A charter-clean, polite, robots-respecting web reader for AI agents — honors robots.txt, identifies honestly, and never bypasses anti-bot/CAPTCHA or paywalls. No account, no API key — free per-call access.

🌐 Live: https://agent-web.foomworks.workers.dev · 🔌 MCP: https://agent-web.foomworks.workers.dev/mcp

What it gives an agent

Give it one publicly reachable URL; get back the page as clean, LLM-ready markdown — title, word count, and the text — without parsing raw HTML yourself. Server-rendered (static) HTML pages. Ideal for summarizing an article, extracting docs, or feeding a page into an LLM.

Connect as an MCP server (recommended)

Streamable HTTP, JSON-RPC 2.0:

https://agent-web.foomworks.workers.dev/mcp
ToolCostReturns
read_urlfreefetch a URL → full LLM-ready markdown (title, word count, markdown)
read_url_previewfreethe first ~600 characters of the markdown — a cheap look before the full pull

Or call it over HTTP

bash
BASE=https://agent-web.foomworks.workers.devcurl -s "$BASE/read?url=https://example.com/&ref=readme"          # full markdowncurl -s "$BASE/read/preview?url=https://example.com/&ref=readme"  # first ~600 chars

Discovery: /.well-known/mcp.json (MCP) · /.well-known/agent-card.json (A2A) · /openapi.json · /policy (Acceptable-Use Policy)

Charter-clean by design (the bright line)

This is a polite reader, not a scraper:

  • Honors the origin's robots.txt for our user-agent — a disallowed path is refused without fetching; an unconfirmable robots.txt (5xx/error) is treated conservatively as disallowed.
  • Identifies honestly with a descriptive user-agent — never spoofs a browser.
  • Read-only GET of the single URL you supply — no crawl, no proxy rotation.
  • Never bypasses anti-bot/CAPTCHA, paywalls, or login walls.
  • SSRF-guarded: private/loopback/link-local/internal hosts are blocked and every redirect hop is re-validated.

You are responsible for your right to fetch the URL you submit.

Limits

  • Static HTML only for now — JavaScript-rendered pages, screenshots, and PDF are a later increment.
  • Body cap ~2 MB, fetch timeout ~12 s.
  • The service holds no keys and never initiates payments.

Install the AgentSkill

SKILL.md is a portable AgentSkill (ClawHub / skills.sh / Claude Skills).

About

agent-web is operated by FOOM — an AI-operated, human-supervised autonomous service. License: MIT.

来源：README.md，提交 24b8d20

工具

5

read_url

Fetch one publicly reachable URL and return clean, LLM-ready markdown (title + page description + word count + markdown). HTML pages are extracted to markdown — including HTML tables, which become GitHub-flavored Markdown tables; URLs pointing straight at a Markdown or plain-text document (raw READMEs, llms.txt, docs) are passed through verbatim. Polite by design: honors the origin's robots.txt for our user-agent, identifies honestly, read-only GET, never bypasses anti-bot/CAPTCHA/paywalls. Free. JavaScript-rendered pages are not supported yet.

输入参数url必填

read_url_preview

Like read_url but returns only the page description + the first ~600 characters of the markdown (title + word count + a truncation flag) — a cheap way to check a page's relevance before pulling the full text. Free.

输入参数url必填

render_preview

Free discovery stub for the screenshot/PDF render lane. Same robots/SSRF guards as read_url. Currently returns a 503 (no charge) until Cloudflare Browser Rendering is provisioned on this account.

输入参数url必填

render_screenshot

PAID (x402): returns x402 payment instructions for a PNG screenshot of a publicly reachable URL, rendered via a real, robots-respecting headless browser. Use render_preview (free) first. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

输入参数url必填widthfullPage

render_pdf

PAID (x402): returns x402 payment instructions for a PDF render of a publicly reachable URL, via a real, robots-respecting headless browser. Currently soft-skips 503 until Cloudflare Browser Rendering is provisioned.

输入参数url必填

版本历史

1
  1. v0.2.1最新Sep 16, 2026

评分

已满足适用检查项的 100%

该评分衡量此收录条目的完整程度，依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价，也不表示服务器安全或当前可用：可连通性是带日期的检查结果，评分也不包含安装量或用户评价。

  • 近 30 天内可连通必需25 分
  • 已声明认证方式必需10 分
  • 已列出工具必需15 分
  • 描述必需15 分
  • 图标10 分
  • 文档15 分
  • 许可证10 分