Web Fetch

作者 alleneubank2921eb8a685a无许可证52 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.

已归档包含脚本Research & Analysis

AI 生成的概览

抓取网页并转换为干净的 Markdown，优先使用原生 Markdown 响应，其次用 CSS 选择器或内置 Bun 脚本。

功能: 该技能从 http/https 网址获取内容并以干净的 Markdown 返回。它先检查响应是否为原生 Markdown，否则使用 html2markdown 配合包含与排除 CSS 选择器提取 HTML，并提供已知文档站点的选择器表。当选择器效果不佳时，运行内置的 Bun 脚本作为备用解析器。
适用场景: 适用于把文档、文章和参考页面抓取为 Markdown 以便阅读或后续处理。适合需要干净文本版本而非原始 HTML 的页面。不适用于仅返回加载占位内容的 JavaScript 渲染页面。
运行要求: 需要 curl、html2markdown 和 Bun，并能访问目标网址的网络。内置的 fetch.ts 脚本需通过 bun install 安装 Bun 依赖。该技能附带可执行脚本。

Web Content Fetching

Fetch web content in this order:

Prefer markdown-native endpoints (content-type: text/markdown)
Use selector-based HTML extraction for known sites
Use the bundled Bun fallback script when selectors fail

Prerequisites

Verify required tools before extracting:

bash

command -v curl >/dev/null || echo "curl is required"command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

command -v curl >/dev/null || echo "curl is required"command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

Install Bun dependencies for the bundled script:

Default Workflow

Use this as the default flow for any URL:

bash

URL="<url>"CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then  curl -sL "$URL"else  curl -sL "$URL" \    | html2markdown \        --include-selector "article,main,[role=main]" \        --exclude-selector "nav,header,footer,script,style"fi

URL="<url>"CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then  curl -sL "$URL"else  curl -sL "$URL" \    | html2markdown \        --include-selector "article,main,[role=main]" \        --exclude-selector "nav,header,footer,script,style"fi

Known Site Selectors

Site	Include Selector	Exclude Selector
platform.claude.com	`#content-container`	-
docs.anthropic.com	`#content-container`	-
developer.mozilla.org	`article`	-
github.com (docs)	`article`	`nav,.sidebar`
Generic	`article,main,[role=main]`	`nav,header,footer,script,style`

Example:

bash

curl -sL "<url>" \  | html2markdown \      --include-selector "#content-container" \      --exclude-selector "nav,header,footer"

curl -sL "<url>" \  | html2markdown \      --include-selector "#content-container" \      --exclude-selector "nav,header,footer"

Finding the Right Selector

When a site isn't in the patterns list:

bash

# Check what content containers existcurl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selectorcurl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line countcurl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

# Check what content containers existcurl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selectorcurl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line countcurl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

Universal Fallback Script

When selectors produce poor output, run the bundled parser:

If already in the skill directory:

Options Reference

bash

--include-selector "CSS"  # Keep only matching elements--exclude-selector "CSS"  # Remove matching elements--domain "https://..."    # Convert relative links to absolute

--include-selector "CSS"  # Keep only matching elements--exclude-selector "CSS"  # Remove matching elements--domain "https://..."    # Convert relative links to absolute

Troubleshooting

Empty output with selectors: The page might be markdown-native. Check headers first:

Wrong content selected: The site may have multiple article/main regions:

html2markdown not found: Install it, then retry selector-based extraction.

bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.

Missing code blocks: Check if the site uses non-standard code formatting.

Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.

来源与署名

来源：alleneubank/claude-code位于.claude/skills/web-fetch提交2921eb8

许可证：无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

Web Fetch

Web Content Fetching

Prerequisites

Default Workflow

Known Site Selectors

Finding the Right Selector

Universal Fallback Script

Options Reference

Troubleshooting

来源与署名

更多来自 alleneubank/claude-code 的技能

React Best Practices

Typescript Best Practices

Zig Best Practices

Python Best Practices

Nix Best Practices

更多Research & Analysis技能

Equity Research

Sector Overview

Idea Generation

Competitive Analysis

Notion Research Documentation

Notion Meeting Intelligence