IPCC AR6 Climate Retrieval

Climate-Risk Analyst Agent

[CI] [Security] [Release] [Python 3.11+] [License: MIT] [LangGraph] [RAG] [Climate risk]

Ask a plain-language question about heat, extreme rain or wind anywhere on Earth, and get back a typed, cited risk report built from live forecast data, 60+ years of ERA5 extremes and IPCC AR6. When a question falls outside what it can check, it refuses instead of guessing.

Live demo: https://climate-risk-agent-714882950125.us-central1.run.app/ (a public Google Cloud Run deployment).

What it does

• Answers for any location: type a place name to geocode it, or enter a latitude and longitude directly.
• Puts a live forecast next to 60+ years of ERA5 annual maxima at that point, and cites the IPCC AR6 pages it used by number.
• Bands the risk on that location's own return-level curve instead of a fixed threshold.
• Reports a confidence that says what it is made of: measured forecast skill at that lead day, the climatology fit, and whether a citation backed the answer.
• Quotes the AR6 Chapter 12 projected change for the reference region containing the point, verbatim.

[Climate-Risk Agent UI: Decision-grade climate intelligence interface with brand logo, interactive deck.gl location map, suggestion chips, and ERA5 GEV risk assessment]

The measurement moat

• ERA5 GEV hazard statistics. Every hazard number comes from a Generalized Extreme Value distribution fitted to 60+ years of ERA5 annual maxima at the query location. Every return level ships with a 90% bootstrap confidence interval.
• Risk levels are local rarity, not a fixed threshold. The band is the return period of the forecast peak on that location's own curve: below the 2-year level is low, 2 to 10 years moderate, 10 to 50 high, at or above 50 severe. The same 46 °C peak can band low where an ordinary year already reaches it and severe where it is unprecedented, off the same code. Each report carries the sentence the band was decided on, naming the bracketing levels and the estimated return period. A location with no fitted climatology falls back to absolute cutoffs and says so rather than passing them off as local.
• Non-stationary GEV. Alongside the stationary fit, a drifting-location GEV checks whether the climate at that location is actually warming, using a likelihood-ratio test to decide. When the trend is real, return levels are reported "effective" at the latest year rather than averaged across six decades. Berlin comes back at about +0.75 °C per decade (p < 0.0001) with effective levels. Delhi comes back stationary, with no significant trend at either Delhi point the app uses (p = 0.46 at the UI's preset Delhi point, p = 0.84 at the geocoded city point about 5 km away), which agrees with the published literature on aerosol masking suppressing South Asian heat trends. The slopes, p-values, exact coordinates and code version are in evals/results/warming-trend-2026-10-21.json, written by scripts/measure_warming_trend.py from a fresh ERA5 1960 to 2022 fit.
• IPCC AR6 RAG with citations that have to hold up. Every citation is checked structurally against the pages actually retrieved for that question. If the model cannot point to a real retrieved page, it refuses rather than cite one anyway.
• Projected change, quoted rather than generated. Each report carries the AR6 WG1 Chapter 12 climatic impact-driver projection for the reference region containing the point: one verbatim Ch.12 sentence, its direction, the IPCC's own calibrated confidence phrase, and the warming level or period it names. All of it is extracted by regex from retrieved chunks, with no model in the loop. When the chapter assesses no such change for that region and hazard, the section is absent and the report says regional projections were not found in the corpus.

The same report, scrolled down: the ERA5 return-level table with bootstrap confidence intervals, the warming-trend banner, and the measured cost and latency for that run.

[ERA5 return levels with 90% bootstrap CIs, the effective-at-2022 warming-trend banner, and per-request cost and latency telemetry]

How it works

A free-text question moves through parsing, geocoding and AR6 region mapping before it reaches the agent. From there a five-node LangGraph agent (plan, call, research, project, synthesize) either produces a typed RiskReport or refuses.

flowchart LR    Q[free-text question] --> PARSE[parse<br/>location · hazard · horizon]    PARSE --> GEO[geocode<br/>Open-Meteo]    GEO --> REGION[map to IPCC AR6 region<br/>reference regions v4]    REGION --> PLAN[agent: plan<br/>scope check]    PLAN -->|unsupported hazard| REFUSE[refusal<br/>valid typed output]    PLAN --> CALL[agent: call<br/>forecast]    CALL --> RESEARCH[agent: research]    RESEARCH --> PROJECT[agent: project<br/>AR6 Ch.12 CID projection]    PROJECT --> SYNTH[agent: synthesize]    SYNTH --> REPORT[RiskReport<br/>typed · cited · grounded]
    subgraph HAZARD [ERA5 to GEV hazard statistics]        ERA5[ERA5<br/>60+ yr annual maxima] --> GEV[stationary + drifting-location GEV]        GEV --> LRT[likelihood-ratio test<br/>is the trend real?]        LRT --> LEVELS[return levels + 90% bootstrap CI<br/>effective at latest year if trend holds]    end    CALL -.-> HAZARD    HAZARD -.-> SYNTH
    subgraph RAG [IPCC AR6 RAG]        HYBRID[BM25 + dense hybrid<br/>RRF fusion] --> VALID[citations validated<br/>against retrieved pages]    end    RESEARCH -.-> RAG    RAG -.-> SYNTH

The UI shows the same run as seven named steps, in this order (agent/progress.py): resolving location, fitting 60 years of ERA5 extremes, checking the question is in scope, fetching the forecast, searching IPCC AR6, reading Chapter 12 projections for your region, writing the cited report.

Evaluation

• Dev set: 60 questions, used to steer development choices like chunking and retrieval configuration. The 15 cid-table questions arrived with the Chapter 12 projection layer, and the set was re-frozen under a new SHA-256.
• Test set: 105 new questions, written after the dev set existed, never used to tune anything, frozen by SHA-256 so neither set can quietly change. 70 are answerable (24 single-page, 22 regional table, 15 multi-page, 9 duplicate region) and 35 are built to be refused (17 premise injection, 10 out-of-scope hazard, 8 out-of-corpus).

Dev-set retrieval, all 60 questions (49 answerable), measured 2026-09-02:

Per slice, hybrid R@3: single-page 92% (n=12), regional table 90% (n=10), multi-page 80% (n=10), premise injection 75% (n=4), cid-table 67% (n=15). cid-table is the weakest slice and is why the headline is 82% across all 60 questions where it was 91% across the original 45.

Dev-set end-to-end, all 60 questions (gemini-2.5-flash, top_k 8, scope guard stage 2 off, measured 2026-09-02):

Citation validity 41/48 (85%). Numeric provenance 42/48 (88%). Grounded refusals 3/4. Cost $0.20 for the run, about $0.0034 per question, p50 latency 5.5 s across 55 model calls with 6 retried rate limits and no failures.

Held-out results (second exposure, on the exact configuration deployed):

• Retrieval, hybrid, on the 70 answerable questions (measured 2026-09-07): R@3 87%, R@5 91%, R@10 96%, that is 61, 64 and 67 of 70.
• End-to-end, all 105 questions (gemini-2.5-flash, top_k 8, measured 2026-07-22): 49 correct answers, 35 correct refusals, 21 false refusals, 0 false answers. All 35 questions built to be refused were refused; the price is 21 of the 70 answerable questions refused as well.
• Citation validity 47/49 correct answers (96%). Numeric provenance 43/49 (88%).
• The IPCC answer step cost $0.29 for the run, about $0.003 per question, p50 generation latency 3.9 s.

Reranking and query rewriting: measured, kept off. Both are wired as HybridRetriever keyword arguments and both default to off. Measured on the 45-question dev set (34 answerable), before the cid-table questions landed:

No arm beat the baseline at any k, and the cheapest of them costs about 100 times the latency. The seams stay in the code because the measurement is what makes leaving them off defensible.

Semantic scope guard, stage 2: measured, kept off. Stage 1 is lexical and runs before any model call. Stage 2 (rag/scope_semantic.py, CRG_SCOPE_STAGE2=off|embed|llm) reads only the questions stage 1 leaves undecided. Measured on the 45-question dev set, as correct answer / correct refusal / false refusal / false answer:

Zero false answers in every arm, and no arm gained anything, so the default stays off. The embed arm's one false refusal carries lexical hazard vocabulary and never reached stage 2. embed is the arm to promote, once a held-out run confirms the matrix holds. The full reasoning is in LIMITATIONS.md.

Every eval artifact records the model that produced it, because a model swap is invisible to a test suite. When the answering model was changed without re-running these evals, the benchmark caught a usefulness regression that the unit tests did not: see adr/0001-answering-model-selection.md.

Refusals are scored on a 4-cell confusion matrix (correct answer, correct refusal, false refusal, false answer). One false answer on that matrix blocks release.

The gate: it is an exit code, not an eyeball. The build fails on any false answer, on headline R@3 falling more than 3 points below the last committed run, or on the hybrid retriever being missing from the artifact, which would mean the run silently measured BM25-only. See DEPLOY.md.

Forecast skill

Every hazard number rests on a forecast, and a peak predicted six days out is not the same claim as one predicted tomorrow. That gap is measured rather than assumed. scripts/measure_forecast_skill.py reads Open-Meteo's Previous Model Runs API, which archives what the model had predicted for each hour 1 to 7 days before it happened, and scores every lead time against the model's own day-0 run over 2024-01-01 to 2025-12-31 at 13 cities on six continents, about 9,200 city-days per lead day.

"Extreme days caught" is how often a day above that location's own 95th percentile was also forecast above it. The frozen numbers sit in tools/forecast_skill_table.json next to their provenance, and tools.forecast_skill.skill_for(hazard, horizon_days) reads them, clamping horizons past day 7, where the archive stops, and flagging them as extrapolated.

The report spends those numbers rather than displaying them. Confidence is min(0.75, 0.3 * w + climatology bonus + 0.1 if IPCC-cited), where w = min(hit rate over lead days 1..L) / hit rate at day 1 is the share of day-1 extreme detection still standing at lead day L. A day-1 heat report keeps the full 0.3 forecast term; a day-7 one keeps 0.56 of it. Every report carries a forecast_skill driver naming the number that did it: "Day-7 temperature forecasts hit the local extreme 47% of the time (GFS Global vs its own day-0 run, 13 cities, 2024-2025), so forecast evidence is down-weighted to 0.56 of its day-1 value." The running minimum in w is deliberate. Precipitation detection measures 1.7% at day 6 and 3.5% at day 7, which is sampling noise on ~458 extreme days rather than a forecast improving with lead time, and the minimum is what keeps confidence monotone, so a longer horizon is never reported as surer.

Three caveats travel with these numbers. The reference series is the model's own latest run rather than station observations, so they are run-to-run consistency errors and a floor on true forecast error. The model is pinned to GFS Global because Open-Meteo's default best_match splices different models across lead offsets and fabricates biases that read like skill: at Delhi it placed the day-1 wind forecast 5.3 km/h off while the day-7 forecast landed within 0.1 km/h. And GFS drops from hourly to 3-hourly output past forecast hour 120, which flattens daily rainfall totals, so the collapse in precipitation detection after day 4 is partly output resolution and not only the forecast missing the storm.

Operations

• One shared cache sits behind the climatology fit, the forecast and the answer cache: Upstash Redis over REST when UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN are set, disk otherwise. tools/cache_backend.py logs the backend it chose, and scripts/prewarm.py fills it before a demo.
• Structured logging and per-request telemetry, measured at the single SDK seam every model call passes through: latency, tokens, retries and cost. Reasoning tokens are counted as billed output, and a model with no price entry reports its cost as unknown rather than as zero. The in-memory event ring is bounded; the JSONL sink keeps the long history.
• Cloud Run idle cost per min-instance shape, the measured 30-day load and the resize plan are in DEPLOY.md.
• Async FastAPI service (POST /report) with per-request API-key access control and a /metrics endpoint.
• Two MCP servers (weather, IPCC RAG) exposing the same tools over the Model Context Protocol.
• CRG_BOOTSTRAP_N sets the GEV bootstrap refits per fit (defaults: 300 stationary, 200 trend-adjusted); it is read at import, is part of the hazard-fit cache key, and trades confidence-interval precision for fit time.
• Docker image measured at 0.31 GB locally. ci.yml runs ruff, mypy and pytest on every push and pull request; evals.yml runs both evals on a v* tag push or on demand, against a pre-built embedding cache downloaded as a release asset, so cutting a release never re-embeds the corpus.

Run it

uv syncuv run streamlit run ui/app.py

Ask in plain language, or use the sidebar to pick any point on Earth: type a place name to geocode it, or enter latitude and longitude directly. Example buttons load the pre-warmed demo cities. A map shows the selected point (display-only) along with its IPCC AR6 region, or a notice when the point has none.

While a request runs, a live progress panel names the step it is on, shows the seconds each one took, and badges any step served from cache with the tier that served it.

The first report for a new place takes roughly 1 to 2 minutes, because 60+ years of ERA5 daily extremes are fetched and the GEV fitted once for that point. Afterwards the fit is cached, Redis when configured and disk otherwise, and repeat visits return in well under a second.

With Docker:

docker build -t climate-risk-agent .docker run -p 7860:7860 -e GEMINI_API_KEY=... climate-risk-agent

The live demo runs on Google Cloud Run. For deployment (Cloud Run, or local Docker and other hosts), see DEPLOY.md.

Use the MCP servers

Both servers speak stdio. From the repo root, point any MCP client at them:

{  "mcpServers": {    "climate-weather": {      "command": "uv",      "args": ["run", "--no-sync", "python", "-m", "tools.weather_mcp"]    },    "climate-ipcc-rag": {      "command": "uv",      "args": ["run", "--no-sync", "python", "-m", "tools.ipcc_mcp"]    }  }}

[search_ipcc called from the MCP Inspector, returning IPCC AR6 excerpts with source file and page number]

Both servers target MCP protocol 2026-07-28 (mcp 2.0.0), the current revision. Every tool is annotated read-only with a human-readable title and an open- or closed-world hint, publishes an outputSchema, and is listed in a deterministic order. Two tests boot each server as a real subprocess and speak the protocol to it, rather than calling the tool functions in-process.

Retrieval is hybrid, so both IPCC tools embed the query and need credentials in the server process: either GOOGLE_GENAI_USE_VERTEXAI=true with GOOGLE_CLOUD_PROJECT, or GEMINI_API_KEY.

A client does not hand the server your shell. It passes a short allow-list of variables (PATH, APPDATA, TEMP, ...) so that a server cannot harvest your secrets, which means a client-launched server starts with no credentials. Supply them either in the client's own env block, or by copying .env.example to .env: the IPCC server reads that at startup and never overrides a value the client did pass.

To explore the tools by hand: uv run mcp dev tools/ipcc_mcp.py.

Published on the official MCP registry

[MCP Registry]

The IPCC RAG server is published on the official MCP registry as io.github.AswaniSahoo/climate-ipcc-rag (v0.1.0), backed by a public OCI image on GHCR (ghcr.io/aswanisahoo/climate-ipcc-rag-mcp:0.1.0). Any MCP client that supports Docker/OCI transport can install it directly from the registry.

[IPCC RAG MCP server listed on the official MCP registry with status Active, showing title, description, and linked repository]

Contributing

Setup, the exact CI commands, the eval release gate, and PR expectations are in CONTRIBUTING.md; participation is covered by the Code of Conduct. Issues use templates for bugs, data correctness, and questions. A wrong return level, a wrong page citation, or a wrong region mapping is a first-class bug here, so please report it as one.

Tech stack

Python, LangGraph, Google Gemini (generation) + gemini-embedding-2 (dense) on Vertex AI (global endpoint), BM25 + dense hybrid retrieval (RRF fusion), Pydantic, FastAPI, Streamlit, MCP Python SDK, scipy, Docker, GitHub Actions.

Limitations

• ERA5 is gridded reanalysis, not station observations. Hazard stats describe an interpolated grid cell near the query location, not a measurement taken there.
• Scope is heat, extreme precipitation, and wind. Anything else should get a refusal, not an answer.
• Coverage is global, but IPCC AR6 regional context is not. Points outside every AR6 land region (open ocean) still get a forecast and hazard stats, with a notice that the regional context is missing.
• The scope guard that keeps out-of-scope hazards away from the LLM is lexical (keyword-based). A paraphrase that avoids the known vocabulary could slip past it.

Data sources and licences

• Forecasts: Open-Meteo, CC BY 4.0.
• ERA5 daily extremes: ERA5 reanalysis, served through the Open-Meteo archive under the same licence.
• Climate assessment: IPCC AR6 WG1 SPM, Chapter 11 and Chapter 12 PDFs, reused for research under IPCC's terms.
• AR6 reference-region polygons: bundled from the IPCC WGI Atlas in tools/ar6/ under CC BY 4.0. Required citation: Iturbide, M., Fernández, J., Gutiérrez, J.M. et al. Implementation of FAIR principles in the IPCC: the WGI AR6 Atlas repository. Scientific Data 9, 629 (2022). https://doi.org/10.1038/s41597-022-01739-y

See LIMITATIONS.md for the full list and SECURITY.md for the threat model. Shipped features and what's next: ROADMAP.md.