Typecast

Typecast

LLM agent that turns messy job descriptions into schema-validated, typed JSON — and measures how often that fails.

Key findings (two Qwen models via OpenRouter, 30 real job descriptions):

• Schema mode makes output valid, not correct. In setup C both models were valid 98.9% of the time, at lower cost than retry, yet end-to-end accuracy was 67.1% (7B) and 84.7% (235B).
• Error-feedback retry fixed the small model, not the large one. Setup B was 98.9% valid for the 7B but 88.9% for the 235B, which failed on jd_002 in all four setup-B runs. A count-aware retry message did not change that.
• No run reached my 90% field-accuracy goal (best: 87.3%), and runs vary. The same setup moved by about 8 points end to end between runs, so a gap between two single runs is not conclusive.

60-second walkthrough:

https://github.com/user-attachments/assets/c4a2572c-88a2-49a3-8e20-176010877154

CLI and package name: soa · MCP server on PyPI: typecast-mcp

[CI]

Results

Two open-weight models through OpenRouter, 30 real job descriptions, four setups. Field accuracy is scored against 20 hand-verified gold labels (see Limitations for how they were made). With 3 runs, the figure is the mean and the brackets are min–max across the runs; with 1 run it is that single run. A = plain prompt, no retry; B = plain prompt + error-feedback retry; C = provider schema mode, no retry; D = schema mode + retry.

How to read it:

• "Mean of 3 runs (min–max)" (B and C) is the mean of three runs on the same 30 documents, with the lowest and highest run in brackets. It is a spread across runs, not an interval estimate. The B runs are the three runs made after the retry-feedback fix described below. The C runs are the first C run plus two repeats. "Single run" (A and D) is one draw each: the repeats show the same setup moving by about 8 points end to end between runs (235B setup B: 72.8% to 80.9%). Each run's own 95% interval (Wilson for the validity columns, bootstrap for the accuracy columns) is in the per-run table at the end of this section; the single runs of A and D have none, because they were made before the interval code existed.
• First-try / final valid: share of the 30 documents whose first reply, or any reply within 3 attempts, passed validation. Avg calls: model calls per document.
• Field accuracy (valid outputs) is measured only on documents that produced a valid posting, so it flatters a setup that fails often (235B setup A: 86.6% field accuracy, but only 70.0% of the labelled documents had a valid output to score). End-to-end accuracy counts a failed document as wrong on every field and is the number to trust when setups differ in validity.
• Cost is the provider-reported cost of one run over the 30 documents. Temperature was not set (the provider default applied, value unknown).

My targets, and whether they were met

I set three targets for setup B before running anything: final validity of 30/30, at most 1.3 calls per document on average, and at least 90% field accuracy. Result, per run (the three runs after the retry fix):

The 90% field-accuracy target was missed by every run of both models, in every setup. The best number anywhere is 87.3% (235B, setup B, first-round run).

How it works

text ──► build prompt (schema + instructions)          │          ▼     LLM call ──► raw string          │          ▼  JobPosting.model_validate_json(raw)     │ ok                     │ ValidationError     ▼                        ▼  log success           log failure (error + raw)  return model          attempt < MAX? ──► append error to messages, retry                              │ no                              ▼                       raise ExtractionFailed (logged)

The schema shown to the model is generated from the Pydantic model (JobPosting.model_json_schema()), never written by hand, and the reply is checked with the same model. A failed reply goes back to the model with the validator's errors, one line per error, for up to 3 attempts. After a valid reply, two deterministic tools tidy the result (canonical skill names, a location check against the document header). They are plain code, not model calls, so they add no LLM calls and the setups below stay comparable. Details: docs/ARCHITECTURE.md.

The four setups: A plain prompt, no retry. B plain prompt, errors fed back (up to 3 attempts). C the provider's strict JSON-schema mode, no retry. D schema mode plus retry.

Why not just use native structured outputs?

Schema mode helped a lot, and it is cheaper. It also does not do the whole job.

What it fixed. Setup C was valid on the first try 98.9% (96.7%-100.0%) of the time for both models, against a single plain-prompt run (A) of 83.3% for the 7B and 70.0% for the 235B. The 30-item cap on skills was the constraint that models actually broke in plain mode; under schema mode the report counts replies that broke a constraint the schema carries, and it was 0 in every C and D run (6 C runs, 2 D runs). C also cost less than B: $0.0076 against $0.0103 for the 7B, $0.0136 against $0.0211 for the 235B (means per run).

What it does not do:

1. It guarantees shape, not meaning. The remaining failures in C were rules no JSON schema can express. Two of the six C runs had one document rejected for duplicate skill names ("LangChain" and "Langchain"; "Python" twice; both rejections are in results/failure_examples.jsonl), which is why C's final validity is 98.9% (96.7%-100.0%) and not 100%. The intern-implies-no-experience rule has the same property (it cannot be stated in a schema); it did not fire in any of the 637 logged attempt lines of this project (the last line of that file records the count), so I have no evidence either way on it.
2. Valid is not correct. Schema mode made the 235B valid 98.9% (96.7%-100.0%) of the time, and its field accuracy is 86.1% (85.6%-87.0%), with end-to-end accuracy 84.7% (82.8%-85.8%): below my 90% goal. The 7B in C is just as valid and gets 68.2% (66.5%-69.8%) field accuracy; its seniority accuracy was 55.0%, 55.0% and 52.6% over the three C runs. Validity says nothing about whether the seniority is right.
3. It may trade recall for validity. The cap is enforced by the model emitting shorter skill lists, and skills are where accuracy is weakest. For the 235B, skills F1 was 66.9%, 72.1% and 65.7% in the three C runs, against 71.4%, 71.6% and 68.3% in the three B runs (means 68.2% vs 70.4%, with overlapping ranges). In the first-round single runs it was 72.1% (A) and 73.8% (B) against 66.9% (C) and 63.4% (D). The repeats show this is a tendency, not a clean effect, and with n=20 labelled documents I would not claim more.
4. It depends on the provider. Both models accepted strict json_schema here, but only because requests were sent with OpenRouter's require_parameters so they could only route to providers that support it. The 7B was always served by one provider; the 235B was split across three.

Most common failure types, and what fixed them

From the plain-prompt runs and the retry runs (counts are failing attempts whose first error was at that field, failures.by_first_loc in the reports):

Almost everything that goes wrong is a count cap, not a formatting problem. In the first 11-document baseline every failure of both models was a too-long skills list (results/2026-11-03_baseline_*.md). The first thing I tried was to make the feedback more specific. The default feedback line is the validator's own wording (for example skills: List should have at most 30 items after validation, not 42). I replaced it for over-long lists with a count and a fix, for example skills: has 39 items, maximum is 30 - remove 9, starting with required=false ones. The result is mixed:

• It did not lift the 235B's final validity in setup B: 86.7% before, then 86.7%, 93.3% and 86.7% across three runs. Each run's interval contains the old figure, so at n=30 no effect is shown.
• The four 235B documents that failed in the first B run (jd_002, jd_003, jd_005, jd_023) passed in 6 of 12 document-runs afterwards (jd_002 0 of 3, jd_003 1 of 3, jd_005 2 of 3, jd_023 3 of 3), but other documents (jd_029, jd_030, jd_004) failed instead. I cannot separate the fix from run-to-run variation. The first attempt is identical before and after the fix, yet first-try validity moved from 83.3% to 76.7%, 73.3% and 80.0%; that spread is run-to-run variation (sampling plus a provider mix I did not log in the first round).
• jd_002 failed for the 235B in all four setup-B runs. Every attempt is in results/failure_examples.jsonl: 42, 31, 31 skills in the first run (before the fix), then 70, 31, 31 and 47, 31, 31 and 44, 31, 31 in the three runs after it. The model goes from 42-70 skills down to 31 on the first retry and then stays at 31. For a 31-item reply the new feedback line is skills: has 31 items, maximum is 30 - remove 1, starting with required=false ones (built by the code; the prompt text itself is not logged). This is the clearest open failure: the model does not make the last cut.

Design decisions

• strict=True on every model. Pydantic's default lax mode quietly turns "3" into 3 and "yes" into True, so a sloppy reply passed and the retry loop had nothing to do. Strict mode rejects both.
• Required-but-nullable fields (company, city, country, min_years_experience). No default: the model has to write null explicitly, so "not stated" is a decision the model made, and a forgotten key is an error rather than a silent null.
• One line per error in the retry prompt, loc: msg (got: <input, clipped to 60 chars>), rendered from ValidationError.errors() instead of str(error). The raw string carries a version-specific prefix, a truncated input_value, and a documentation URL, which are noise for a model. The one exception is an over-long list, which gets the count-aware line shown above. I did not run a blind-retry control (retry with no error text), so I have no measurement of feedback versus plain resampling; setup A against B is retry-with-feedback against nothing.
• Tools are deterministic post-processing, not model-called. Canonical skill names and a location check run after validation as plain code. That keeps every setup at one prompt and the retry/attempt/cost counts clean, and the model's own output is kept (raw_posting) so validity is measured on what the model actually said. Letting the model call the tools is a separate experiment I have not run.
• API errors are separate from validation errors. A 429 or 5xx is retried with backoff (2, 4, 8 s) and never uses up one of the 3 validation attempts.
• The 30-skill cap stays. The first plain-prompt baseline showed every failure was this one constraint, so I kept it as the test of what retries and schema modes can do rather than loosening it to make numbers look better.

Limitations

• 30 job descriptions, 20 gold labels. One document moves a validity rate by 3.3 points and a field accuracy by 5 points; the intervals above are wide for that reason (for example 235B setup B final validity 93.3% [78.7-98.2] in one run). Five of the 30 are non-technical or semi-technical roles, kept on purpose as an out-of-domain slice; two of them (jd_012, jd_019) are among the 20 labelled.
• The gold labels were drafted by a model (Claude) and then checked by me. I verified every field and made 17 corrections (docs/label_corrections.log). One person, one pass, so any systematic mistake of mine is in the labels. The labelling rules were written into the schema's field descriptions after the first baseline, so the models and the labels follow the same rules; runs on the earlier prompt are not comparable.
• Noise floor on accuracy, from a blind check. On 5 other documents (jd_004, 005, 008, 014, 015, none of them in the gold set) I labelled blind and compared against Claude's sealed labels written beforehand, with the same per-field metrics as the eval (my labels as the reference): title 80.0%, company 100.0%, seniority 80.0%, mode 80.0%, salary_mentioned 100.0%, min_years_experience 100.0%, city 80.0%, country 100.0%, skills F1 61.0% (precision 52.9%, recall 72.0%), overall 86.8% (results/label_agreement.md). Five documents is a sanity check, not a rate: one disagreement moves a field by 20 points. It does suggest that on skills two careful labelers disagree about as much as the models disagree with the gold labels, and that the 235B's roughly 86% field accuracy is close to the level two labelers reach with each other, so the remaining gap to my 90% target may partly be label noise rather than model error. I have not shown that.
• Runs are from one day, two models of one family, one provider layer. Temperature was not set. The 7B was always routed to one provider; the 235B to three, and the provider was not logged in the first round, so provider mix and sampling noise are mixed into the run-to-run spread.
• The post-processing tools were not tuned against the final labelling rules, so the location check and the labels can disagree on some documents.
• Setup D is a single run per model; only B and C were repeated.
• Data. The 30 job descriptions in data/jds/ were copied from public postings (LinkedIn 13, Work at a Startup 11, Wellfound 6, counted from data/jds/manifest.csv) for evaluation only; I will remove any of them on request. Five rows have no source URL recorded in manifest.csv: jd_001, jd_002, jd_003, jd_004 and jd_010 (all LinkedIn; their notes say "URL not recorded").

Run it yourself

Needs Python 3.11+ and uv.

git clone https://github.com/nilayp-codes/typecast.gitcd typecastuv synccp .env.example .env        # then set OPENROUTER_API_KEY in .env (never commit it)
uv run soa extract data/jds/jd_001.txt                 # the JSON on stdout, a status line on stderruv run soa extract data/jds/jd_001.txt --json-mode     # provider schema modeuv run soa extract data/jds/jd_001.txt > out.json
uv run soa eval --model openrouter/qwen/qwen-2.5-7b-instruct --setup B --limit 5uv run soa eval --model openrouter/qwen/qwen3-235b-a22b-2507 --setup all

Known issue: on Windows, clone into a short path, because LiteLLM writes a cache file under .venv and a very deep path exceeds Windows' path-length limit (soa extract then fails with APIConnectionError ... [Errno 2]).

soa eval makes real model calls: it prints the plan and the estimated number of calls and asks before sending anything (--yes skips the question). In my runs one setup over the 30 documents cost between $0.0076 and $0.0234. It saves results/<date>_<model>_<setup>.json and appends a row to results/summary.md; --resume <eval id> continues an interrupted run. To combine repeated runs: uv run python scripts/aggregate_runs.py results/<files>.json (writes results/aggregate.md). The 30 job descriptions are in data/jds/ and the gold labels in data/labels/.

Tests and type checks (no real model is ever called in tests):

uv run pytestuv run mypy --strict src/ packages/typecast-mcp/src/

Use it from an AI agent (MCP server)

typecast-mcp is a small MCP server that checks a job-posting JSON against the JobPosting schema and returns one-line errors an agent can act on. It has no LLM, needs no API key and makes no network calls. It is its own package (packages/typecast-mcp/) with two dependencies, pydantic and mcp, so installing it does not pull in LiteLLM or the rest of the eval harness.

Two tools:

• get_job_posting_schema: the JSON Schema of a job posting. It does not carry every rule (duplicate skill names and the intern/experience rule are only checked by the validator).
• validate_job_posting: takes one argument, posting, the whole job-posting JSON as ONE string. Returns valid, a list of error lines (field: problem (got: value)), and the parsed posting when valid.

Install (needs uv). Claude Desktop, Cursor and other clients:

{  "mcpServers": {    "typecast": {      "command": "uvx",      "args": ["typecast-mcp"]    }  }}

Claude Code: claude mcp add typecast -- uvx typecast-mcp. On Windows, if the client cannot find uvx, put the full path to uvx.exe in command (find it with where.exe uvx). The Microsoft Store build of Claude Desktop does not read %APPDATA%\Claude\claude_desktop_config.json; open Settings → Developer → Edit Config to open the file it does read. Without uv: pip install typecast-mcp and use typecast-mcp as the command.

From a clone. The repo ships a .mcp.json that runs the server from the working tree (uv run typecast-mcp). Open the folder in Claude Code, approve typecast when asked, and check it with /mcp. Approving runs uv run in this folder, which installs the locked dependencies and starts the server. Only approve it in a clone you trust.

A real example. Qwen3-235B's reply on jd_002 had 70 skills. An agent checked it with validate_job_posting and got one line back:

skills: has 70 items, maximum is 30 - remove 40, starting with required=false ones

The agent fixed the posting in one round. This was one run with Claude Sonnet 5.5, not a benchmark.

Demo in Claude Desktop (errors planted for the demo). Claude calls validate_job_posting, gets two error lines, fixes them and re-validates. Note that it had to guess the years to make the posting pass:

https://github.com/user-attachments/assets/4190ef12-fc7c-46f8-ad48-dd128d80aa54

What it does not check: whether the values are correct. A posting can be valid and still have the wrong seniority or the wrong skills; see Why not just use native structured outputs?.

Details, limits and design decisions: docs/MCP.md.

What I'd do next

1. A validator-only MCP server: done, see Use it from an AI agent.
2. Publish on PyPI: done for the MCP server, typecast-mcp.
3. Fix the remaining failure the data points at: the model that cannot make the last cut on jd_002, for example by telling it which items to drop. Re-measure with temperature fixed and the provider pinned, so repeats differ less.
4. A second labeler for the gold set, a bigger document set, and repeats of setup D, so the field-accuracy numbers stop being limited by n=20.

License

MIT. See LICENSE.