Web Fetch

by alleneubank2921eb8a685aNo license52 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago

Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.

ArchivedIncludes scriptsResearch & Analysis

AI-generated overview

Fetches web pages and converts them to clean markdown using markdown-native responses, CSS selectors, or a bundled Bun script.

What it does: The skill retrieves content from http/https URLs and returns it as clean markdown. It first checks whether the response is markdown-native, otherwise extracts HTML with html2markdown using include and exclude CSS selectors, including a table of selectors for known documentation sites. When selectors produce poor results, it runs a bundled Bun script as a fallback parser.
When to use it: Use it to pull documentation, articles, and reference pages into markdown for reading or further processing. It suits pages where a clean text version is needed rather than raw HTML. It is not intended for JavaScript-rendered pages that only return loading placeholders.
Requirements: Requires curl, html2markdown, and Bun, plus network access to the target URLs. The bundled fetch.ts script needs its Bun dependencies installed with bun install. It ships an executable script.

Web Content Fetching

Fetch web content in this order:

Prefer markdown-native endpoints (content-type: text/markdown)
Use selector-based HTML extraction for known sites
Use the bundled Bun fallback script when selectors fail

Prerequisites

Verify required tools before extracting:

bash

command -v curl >/dev/null || echo "curl is required"command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

command -v curl >/dev/null || echo "curl is required"command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

Install Bun dependencies for the bundled script:

Default Workflow

Use this as the default flow for any URL:

bash

URL="<url>"CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then  curl -sL "$URL"else  curl -sL "$URL" \    | html2markdown \        --include-selector "article,main,[role=main]" \        --exclude-selector "nav,header,footer,script,style"fi

URL="<url>"CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
if echo "$CONTENT_TYPE" | grep -q "markdown"; then  curl -sL "$URL"else  curl -sL "$URL" \    | html2markdown \        --include-selector "article,main,[role=main]" \        --exclude-selector "nav,header,footer,script,style"fi

Known Site Selectors

Site	Include Selector	Exclude Selector
platform.claude.com	`#content-container`	-
docs.anthropic.com	`#content-container`	-
developer.mozilla.org	`article`	-
github.com (docs)	`article`	`nav,.sidebar`
Generic	`article,main,[role=main]`	`nav,header,footer,script,style`

Example:

bash

curl -sL "<url>" \  | html2markdown \      --include-selector "#content-container" \      --exclude-selector "nav,header,footer"

curl -sL "<url>" \  | html2markdown \      --include-selector "#content-container" \      --exclude-selector "nav,header,footer"

Finding the Right Selector

When a site isn't in the patterns list:

bash

# Check what content containers existcurl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selectorcurl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line countcurl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

# Check what content containers existcurl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selectorcurl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line countcurl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

Universal Fallback Script

When selectors produce poor output, run the bundled parser:

If already in the skill directory:

Options Reference

bash

--include-selector "CSS"  # Keep only matching elements--exclude-selector "CSS"  # Remove matching elements--domain "https://..."    # Convert relative links to absolute

--include-selector "CSS"  # Keep only matching elements--exclude-selector "CSS"  # Remove matching elements--domain "https://..."    # Convert relative links to absolute

Troubleshooting

Empty output with selectors: The page might be markdown-native. Check headers first:

Wrong content selected: The site may have multiple article/main regions:

html2markdown not found: Install it, then retry selector-based extraction.

bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.

Missing code blocks: Check if the site uses non-standard code formatting.

Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.

Source and attribution

Source:alleneubank/claude-codein.claude/skills/web-fetchat commit2921eb8

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Web Fetch

Web Content Fetching

Prerequisites

Default Workflow

Known Site Selectors

Finding the Right Selector

Universal Fallback Script

Options Reference

Troubleshooting

Source and attribution

More from alleneubank/claude-code

React Best Practices

Typescript Best Practices

Zig Best Practices

Python Best Practices

Nix Best Practices

More in Research & Analysis

Equity Research

Sector Overview

Idea Generation

Competitive Analysis

Notion Research Documentation

Notion Meeting Intelligence