Legal Data Hunter

World Wide Law

Open-source collection scripts for open legal data from 110+ countries.

Every country publishes its laws, court decisions, and regulations online -- but in different formats, behind different APIs, with different access rules. World Wide Law is building the open infrastructure to collect, normalize, and make all of it searchable.

All sources in this repository are open data -- publicly available legal information from official government portals, APIs, and bulk download endpoints. We always prefer API and bulk access over web extraction.

Live Dashboard & API

• Dashboard: legaldatahunter.com -- track coverage, explore sources, submit feedback
• Search API: Available at legaldatahunter.com -- search across 16M+ indexed legal documents

What's Here

This repository contains 960+ collection scripts across 110+ countries that download and normalize open legal data from government portals worldwide. Each script follows a standard interface so that any developer can run, test, or improve it. Some sources are marked as blocked (CAPTCHA, IP restrictions, etc.) -- their scripts are included so developers can review and potentially contribute fixes.

sources/  FR/LegifranceCodes/     # French consolidated legal codes (API)  DE/GesetzeImInternet/   # German federal laws (bulk XML)  IT/NormattivaLegislation/ # Italian legislation (API)  ES/BOE/                 # Spanish official gazette (API)  ... (110+ countries)

Quick Start

# Clone the repogit clone https://github.com/worldwidelaw/legal-sources.gitcd legal-sources
# Install dependenciespip install -r requirements.txt
# Check project statuspython runner.py status
# Test a specific sourcepython runner.py sample FR/LegifranceCodes
# See what needs workpython runner.py next

How It Works

Per-Source Structure

Every source lives in sources/{COUNTRY_CODE}/{SourceName}/ and contains:

Two Data Models

Legislation (mutable): Laws get amended. Same ID, new content. Strategy: upsert with version tracking.

Case law (immutable): Court decisions don't change after publication. Strategy: append-only with dedup.

Standard Output Schema

Every script normalizes documents to a common schema:

• _id -- Unique identifier
• _source -- Source identifier (e.g., FR/LegifranceCodes)
• _type -- legislation or case_law
• title -- Document title
• text -- Full text content
• date -- Publication or decision date
• url -- Link to the original source

Architecture

legal-sources/  manifest.yaml          # Master inventory: all sources + status  runner.py              # CLI: run, test, and manage collection scripts  common/                # Shared libraries    base_scraper.py        Base class all scripts inherit from    http_client.py         HTTP client with retries + caching    rate_limiter.py        Token bucket rate limiter    storage.py             JSONL storage with deduplication    validators.py          Schema validation  templates/             # Templates for new sources    scraper_template.py    Boilerplate for bootstrap.py    config_template.yaml   Boilerplate for config.yaml    retrieve_template.py   Boilerplate for retrieve.py  sources/               # One directory per data source    {CC}/{Source}/          (see per-source structure above)

Coverage

Track live progress on the dashboard.

Contributing

We welcome contributions from developers, legal researchers, and especially governments who want their open legal data included.

Who can contribute?

Submit a data source (no coding required):

• Open a "New Source" issue and tell us about your country's legal data portal

Fix or improve a collection script:

• See CONTRIBUTING.md for the full guide

Report a problem:

• Data quality issue -- missing or incorrect data
• Bug report -- broken script

Good first issues: Browse label:good-first-issue for approachable starting points.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). You are free to use, modify, and distribute this software, provided that any modified versions made available over a network also make their source code available under the same license.

Commercial Licensing: If you wish to use this software without the AGPL-3.0 obligations (e.g., in a proprietary product or SaaS), commercial licenses are available. Contact [email protected] for details.