ABS Data API — cartography and front door
This project measured what the Australian Bureau of Statistics Data API actually serves — every dataflow, by retrieval —
and exposes the result as a typed front door: an MCP server and an HTTP API
that only ever offer what exists.
[The ABS Data API cartography: unmasking the 4.21% reality — a comic-style map of the findings and the front door]
What was found
- 621,536,714 series exist across 1,227 dataflows. ABS's metadata implies
14,761,557,818 — a density of 4.21%, overstated 23.8×.
- The per-dimension metadata is exact: every advertised option exists, and
nothing real is unadvertised. The entire overstatement is combinatorial.
- 81.0% of series sit in flows whose sparsity is principled and verified in
the keys — almost entirely hierarchical geography encoded as separate
dimensions (
STATE is a function of REGION in 664 flows). The remaining
19% is not reducible to a single-dimension dependency.
detail=serieskeysonly — the endpoint that should answer "what exists" — is
broken (HTTP 200, malformed JSON). Eight of fifteen documented behaviours do
not conform. See reports/.
DESIGN.md is the full record: verified API findings, every decision and why,
and the corrections made along the way.
Layout
Running it
Rate limits, per client address: 600 requests/min overall; 120/min for data
pulls (each is a live ABS call); 20/min for Code Mode execute. Over the limit,
HTTP doors answer 429 with Retry-After; an MCP tool call gets a tool error.
Using the SDK from TypeScript:
The probe is resumable: rerunning skips flows already recorded as complete. For
the largest census flows run with NODE_OPTIONS=--max-old-space-size=20480.
Gotcha: wrangler keys the local D1 state by database_id. Changing the id in
wrangler.toml (e.g. after d1 create) gives wrangler dev a fresh, empty
local database — re-run the local import.
Principles
- Observed-only. Declared metadata steers the crawler and feeds the defect
report; it never enters what a caller sees. Availability is empirical: observed
options from the catalogue, exact combination checks live against ABS's
availableconstraint (which agrees with the probe on every dimension of every
flow).
- One contract, derived doors. Every crawl regenerates one contract; the MCP
verbs, the HTTP routes, the OpenAPI document and the docs all read or render
it. Nothing hardcodes the surface, so nothing drifts.
- Only calls that return data. A selection is verified before it is
forwarded; an invalid one comes back with the observed valid options — as an
MRTR
input_required on MCP, a typed 422 on HTTP.