Crawler Log Triage

dev.workers.pathwren.www由 www.pathwren.workers.dev 发布官方发布者v1.7.0更新于 Oct 24, 2026

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

发布方自己的服务:Agent Discovery Doctor - llms.txt, agent card, mcp.json

已验证Streamable HTTP可网页运行开发者工具安全与监控

概览

AI 生成的概览

让助手分析网站访问日志,识别爬虫与运营者、找出冒充者,并生成 robots.txt 或 WAF 规则。

可以做什么
该服务分析你粘贴的网页服务器访问日志,逐行判断爬虫、运营者、用途与成本。它可以汇总各运营者的请求数与占比、找出声称是爬虫但来源地址不在其公布范围内的冒充者,并根据日志生成 robots.txt 及 nginx、Caddy、Cloudflare、HAProxy 或 Apache 的 WAF 规则。另有工具返回本主机自身日志的示例分析,以及报告你这次请求的 user-agent 分类。
适用场景
当你需要判断网站访问日志中哪些流量来自爬虫、各运营者的请求量与封禁成本,或想找出冒充已公布地址范围的爬虫时使用。也适合用来根据日志生成 robots.txt 或 WAF 规则。
运行要求
需要可连接远程 MCP 端点的网络访问;不需要 API 密钥、账号或环境变量。
安装前请注意
部分工具会返回本主机自身的请求日志分析,并记录你的请求与 user-agent;粘贴的日志可能含 IP 地址等访客数据,发送前请自行评估。生成的 WAF 规则仅依据 user-agent 判断,只能拦截守规矩的客户端。

安装

在 SourceWeft 中

  1. 打开 控制台中的 Crawler Log Triage,将其添加到工作区。
  2. 为需要使用其工具的对话启用该服务。

Web executable,通过 Streamable HTTP。 远程服务在工作区中配置后即可从网页运行时运行。

交给智能体安装

把这段提示词粘贴到 Claude Code、Codex、Cursor 等智能体,或 SourceWeft 对话中。智能体会读取该服务器的安装指南,先向你展示发布者、端点和登录方式,经你同意后再把它添加到你的客户端。密钥不会写进任何命令。

阅读 https://sourceweft.com/mcp/dev.workers.pathwren.www%2Fcrawler-log-triage/install.md,并按其中说明把这个 MCP 服务器接入我的客户端。修改任何配置之前,先告诉我它的发布者、端点和登录方式,等我确认。不要把 API 密钥写进命令或我会提交的文件;问我密钥应从哪里获取。

查看智能体所依据的安装指南

其他 MCP 客户端

把它添加到你客户端的 mcpServers 配置中。

{
  "mcpServers": {
    "crawler-log-triage": {
      "type": "http",
      "url": "https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/triage"
    }
  }
}

工具

9

no_arguments_triage_this_hosts_own_crawler_log

No arguments: triage this host's own published crawler log

TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} to https://www.pathwren.workers.dev/mcp/triage — the answer is the triage of THIS host's own published request log — every operator in it run through the same parser, the same crawler index and the same operator-prefix verification that triage_log applies to a file you paste, rolled up by operator, by category and by crawler, with the share no index entry matches at all and the browser-shaped strings named separately. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. The other zero-argument call on this server is triage_my_request, same empty arguments, which answers your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","argumen

无需输入参数。

triage_my_request

Triage my own request (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and the answer is your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers your own request triaged as one line of an access log, by the same parser and the same index that read a whole file — its six siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev; whoami is answered by all seven and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}} returns the single log line built from your address and your user-agent, the triage row for it, and the same rollup triage_log returns for a whole file.

无需输入参数。

whoami

Who is calling? — the one call that needs nothing from you

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/triage and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: the request you just made, run through triage_log as a single log line, naming the crawler it identifies, its operator and category, and whether your address verifies against that operator's published prefixes. Takes no arguments. Safe to call. Deterministic. Touches no third party. Nothing is fetched to build it: every fact comes from the headers on your own request or from a file this host already publishes. Only this server answers the last part — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev. Example: arguments={} returns your user-agent, your address, the class we book you as, whether we have seen you here before, and your own request triaged as one line of an access log, by the same parser and the same index that read a whole file.

无需输入参数。

example

Run this server's worked example (no arguments)

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — triage_log, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — triage_log on three lines in three of the five formats this parser accepts: an address Google publishes, an address in no published range claiming to be the same crawler, and a `uniq -c` count — all of it constants in this file, checked against our own mirror, with nothing fetched from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs triage_log with {"log":"66.249.66.1 Googlebot/2.1\n203.0.113.9 Googlebot/2.1\n412 GPTBot/1.2"} and returns its real answer.

无需输入参数。

triage_log

Triage a paste of log lines

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Paste log lines — combined, JSON, `uniq -c`, `<ip> <ua>` or bare user-agents, auto-detected — for per-line crawler, operator, purpose and cost of blocking, a rollup by operator and category, and every line whose address contradicts its claim. Log text, never a URL. Example: log='66.249.66.1 Googlebot/2.1' returns Googlebot, Google, search, verified.

输入参数log必填limitdetail

find_impersonators

Find the lines that are lying

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Only the lines claiming a crawler whose operator publishes address ranges, from an address in none of them — 1997 IPv4 and 1062 IPv6 prefixes, 15 sources. Reverse-DNS operators come back with the command to run: this server makes no outbound request. Example: log='203.0.113.9 Googlebot/2.1' returns one impersonation.

输入参数log必填

summarize_by_operator

Roll a log up by operator and category

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Aggregate only: who crawled you, how many requests each, what share, which category, and what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. Example: log='412 GPTBot/1.2' returns OpenAI, 412 requests, 100%, ai-training.

输入参数log必填

robots_from_log

robots.txt from a log

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — A robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only.

输入参数log必填stance

waf_ruleset_from_log

WAF ruleset from a log

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — nginx, Caddy, Cloudflare, HAProxy or Apache rules for only the crawlers in your log. The reply warns that a UA rule stops only an honest client, and that impersonation is an address problem needing the published prefixes as an allowlist. Example: log='412 GPTBot/1.2', target='nginx', scope='ai-training'.

输入参数log必填scopeactiontarget

版本历史

1
  1. v1.7.0最新Sep 16, 2026

评分

已满足适用检查项的 83%

该评分衡量此收录条目的完整程度,依据注册中心条目、端点与公开来源计算得出。它不是对服务器本身的评价,也不表示服务器安全或当前可用:可连通性是带日期的检查结果,评分也不包含安装量或用户评价。

  • 近 30 天内可连通必需25 分
  • 已声明认证方式必需10 分
  • 已列出工具必需15 分
  • 描述必需15 分
  • 图标10 分
  • 文档15 分

    请声明包含 README 的 GitHub 仓库,或在 OAuth 受保护资源元数据中提供 resource_documentation。

  • 许可证不适用

    托管服务按其提供方的条款使用,因此这里不要求许可证。此检查项不计入百分比。

其他 Workers 集成