Web Scraper

guia-matthieu/clawfu-skills/skills/automation/web-scraper

by guia-matthieu4108f5c010ccaa8b5707fa05ca1d888ae167e548MITListed Oct 9, 2026Updated Oct 9, 2026

Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

Includes scriptsData & Analytics

AI-generated overview

Extracts structured data, links, emails and page elements from websites using a Python scraping script.

What it does: Provides a command-line Python script that fetches web pages and extracts content via CSS selectors, collects links, harvests email addresses, and pulls structured fields such as article or product data. Output is printed or written to files like JSON. It also documents CSS selector syntax and ethical scraping practices.
When to use it: Use it for collecting competitor pricing, product listings, contact information, research data, or for monitoring changes on web pages.
Requirements: Python with beautifulsoup4, requests, pandas, click and lxml installed; network access to the target sites. Ships executable scripts (scripts/main.py, scripts/requirements.txt).

Web Scraper

Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.

When to Use This Skill

Competitor research - Scrape pricing, features, positioning
Lead generation - Extract contact info from directories
Content audit - Pull headings, links, meta data
Price monitoring - Track competitor pricing changes
Data collection - Gather research data from multiple sources

What Claude Does vs What You Decide

Claude Does	You Decide
Structures analysis frameworks	Strategic priorities
Synthesizes market data	Competitive positioning
Identifies opportunities	Resource allocation
Creates strategic options	Final strategy selection
Suggests implementation approaches	Execution decisions

Dependencies

Commands

Scrape Elements

bash

python scripts/main.py scrape https://example.com --selector "h1,h2,p"python scripts/main.py scrape https://example.com --selector ".product-price"

python scripts/main.py scrape https://example.com --selector "h1,h2,p"python scripts/main.py scrape https://example.com --selector ".product-price"

Extract Links

bash

python scripts/main.py links https://example.compython scripts/main.py links https://example.com --internal-only

python scripts/main.py links https://example.compython scripts/main.py links https://example.com --internal-only

Extract Emails

bash

python scripts/main.py emails https://example.compython scripts/main.py emails https://example.com --depth 2

python scripts/main.py emails https://example.compython scripts/main.py emails https://example.com --depth 2

Extract Structured Data

bash

python scripts/main.py structured https://example.com/article --schema articlepython scripts/main.py structured https://example.com/product --schema product

python scripts/main.py structured https://example.com/article --schema articlepython scripts/main.py structured https://example.com/product --schema product

Examples

Example 1: Scrape Competitor Pricing

bash

python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"
# Output:# Extracted 6 elements# 1. Starter - $29/mo# 2. Pro - $99/mo# 3. Enterprise - Contact us

python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"
# Output:# Extracted 6 elements# 1. Starter - $29/mo# 2. Pro - $99/mo# 3. Enterprise - Contact us

Example 2: Extract Article Content

bash

python scripts/main.py structured https://blog.example.com/post --schema article
# Output: article_data.json# {#   "title": "How to Scale Your Startup",#   "author": "Jane Doe",#   "date": "2024-01-15",#   "content": "...",#   "word_count": 1523# }

python scripts/main.py structured https://blog.example.com/post --schema article
# Output: article_data.json# {#   "title": "How to Scale Your Startup",#   "author": "Jane Doe",#   "date": "2024-01-15",#   "content": "...",#   "word_count": 1523# }

CSS Selector Reference

Selector	Description	Example
`tag`	Element type	`h1`, `p`, `div`
`.class`	Class name	`.price`, `.title`
`#id`	Element ID	`#main-content`
`tag.class`	Tag with class	`div.product`
`tag[attr]`	Has attribute	`a[href]`
`parent > child`	Direct child	`ul > li`
`tag1, tag2`	Multiple	`h1, h2, h3`

Ethical Scraping Guidelines

Check robots.txt - Respect site's scraping policy
Rate limit - Don't overload servers (1-2 req/sec)
Identify yourself - Use descriptive User-Agent
Cache requests - Don't re-scrape unchanged pages
Terms of Service - Check if scraping is allowed

Skill Boundaries

What This Skill Does Well

Structuring strategic analysis
Identifying market opportunities
Creating strategic frameworks
Synthesizing competitive data

What This Skill Cannot Do

Replace market research
Guarantee strategic success
Know proprietary competitor info
Make executive decisions

Related Skills

competitor-monitor - Monitor competitor changes
pdf-extractor - Extract from PDFs

Skill Metadata

Mode: centaur

yaml

category: automationsubcategory: data-extractiondependencies: [beautifulsoup4, requests, pandas]difficulty: intermediatetime_saved: 5+ hours/week

category: automationsubcategory: data-extractiondependencies: [beautifulsoup4, requests, pandas]difficulty: intermediatetime_saved: 5+ hours/week

Source and attribution

Source:guia-matthieu/clawfu-skillsinskills/automation/web-scraperat commit4108f5c

License: MIT

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Web Scraper

Web Scraper

When to Use This Skill

What Claude Does vs What You Decide

Dependencies

Commands

Scrape Elements

Extract Links

Extract Emails

Extract Structured Data

Examples

Example 1: Scrape Competitor Pricing

Example 2: Extract Article Content

CSS Selector Reference

Ethical Scraping Guidelines

Skill Boundaries

What This Skill Does Well

What This Skill Cannot Do

Related Skills

Skill Metadata

Source and attribution

More from guia-matthieu/clawfu-skills

Yc Pitch Deck

Conversion Copywriting

Persuasion Principles

Sound Design Film

Report Generator

Whisper Transcription

More in Data & Analytics

Audit Xls

Comps Analysis

Audit Xls

Audit Xls

Comps Analysis

Audit Xls