Ticket AI

ticket-ai

[release] [MCP tools] [CLI commands] [trackers]

[tests] [coverage] [python] [license]

An MCP server and CLI that measures how your team actually writes tickets, and holds new ones to that. Not this ticket is bad — 31 of the 40 tickets that shipped here have an acceptance-criteria section, and this one does not.

No 'Akzeptanzkriterien' section. 31 of the 40 exemplar tickets (78%) have one.

Most ticket advice is free and therefore ignored. "Add acceptance criteria", "include steps to reproduce" — everyone has heard it, nobody changed anything. A count of what already happened in your own project is harder to wave away, and it is not an opinion about tickets.

There is no model in the loop for any of that. Learning the house style, finding related tickets and measuring a draft are counting, and run with no key and no network beyond your tracker. Writing a ticket needs a model, so that part is opt-in and which one is your choice. What it found when it was run against real boards →

Install

uv tool install ticket-ai-mcp

Configuration is environment variables only — a token passed as --token ends up in your shell history and in the process list.

export TICKET_AI_TRACKER=gitlabexport TICKET_AI_PROJECT=acme/shop        # or the numeric idexport TICKET_AI_GITLAB_URL=https://gitlab.example.comexport TICKET_AI_GITLAB_TOKEN=...         # read_api scope is enough

The token is optional on a public project. TICKET_AI_GITLAB_URL=https://gitlab.com with no token reads any public board, which is the quickest way to see what this does before pointing it at your own instance.

# Jira — Cloud or self-hosted Server / Data Center. Which one you are on is# detected from the instance; you do not have to say.export TICKET_AI_TRACKER=jiraexport TICKET_AI_PROJECT=PROJexport TICKET_AI_JIRA_URL=https://acme.atlassian.net
# Leave the credentials unset for a public board — plenty answer without any.# Cloud: the token comes from id.atlassian.com, and is not the password.export [email protected]export TICKET_AI_JIRA_TOKEN=...# Self-hosted: a personal access token on its own, sent as a Bearer.export TICKET_AI_JIRA_TOKEN=...# Only if detection gets it wrong: cloud | server | auto (the default)export TICKET_AI_JIRA_API=server
# GitHubexport TICKET_AI_TRACKER=githubexport TICKET_AI_PROJECT=acme/shopexport TICKET_AI_GITHUB_TOKEN=...

mcp-name: io.github.syrian963/ticket-ai-mcp

One command

ticket-ai learn      # mine the tracker, cache the profileticket-ai style      # what it learned

learn takes a minute or two: ranking needs each ticket's comments and linked merge requests, which is an extra request or two per ticket. It caches to TICKET_AI_CACHE_DIR if you set one and to .ticket-ai/ otherwise, so it happens once rather than once per review.

Then the rest:

ticket-ai learn --from '#412,#98'                  # or name the good ones yourselfticket-ai context 'export is broken on mobile'     # what already existsticket-ai gaps                                     # declared template vs what arrivesticket-ai draft --title '...' --file draft.md      # check one before creating itticket-ai review '#42'                             # measure one that existsticket-ai open                                     # every open ticket, worst first

draft is the one worth building a habit around. Checking a ticket after you create it puts the review past the point of no return: the board has already been notified and every fix is now an edit with a history.

$ ticket-ai draft --title 'Filter kaputt' --file draft.mdAlignment with house style: 33% over 3 checks
### MEDIUM - The description is 116 characters.The shortest quarter of tickets that shipped here start at 639; the median is 1079.
### MEDIUM - The ticket has no labels.62% of the exemplars are labelled.

--from is taken as given: no filtering, no scoring against your choices. Name a ticket with a three-word description and that is your answer about how this team writes tickets, and the profile will say so.

As an MCP server

{  "mcpServers": {    "ticket-ai": {      "command": "uvx",      "args": ["ticket-ai-mcp"],      "env": {        "TICKET_AI_TRACKER": "gitlab",        "TICKET_AI_PROJECT": "acme/shop",        "TICKET_AI_GITLAB_URL": "https://gitlab.example.com",        "TICKET_AI_GITLAB_TOKEN": "..."      }    }  }}

Eight tools, all read-only: learn_conventions, house_style, ticket_template, ticket_context, template_gaps, review_draft, review_ticket, review_open_tickets.

Add TICKET_AI_REPO to the env block if the checkout you want searched is not the assistant's working directory.

Writing a ticket this way

The path to reach for if you already use Claude Code: no key, no compose, no second model call. Ask for a ticket and the assistant does five things, three of them here:

1. ticket_template — the shape: which sections, how long, what language, which labels.
2. ticket_context — what exists: related tickets, the files their merge requests changed, the files in the checkout that mention it.
3. It reads those files. ticket_context runs a text search, not an analysis; it says where to look, it does not save you looking.
4. It writes the ticket.
5. review_draft — measures what it wrote, and fixes what that finds before showing you anything.

Where the "AI" is

Counting cannot produce a paragraph of German, so writing a ticket needs a model. Everything else needs nothing.

export TICKET_AI_WRITER=ollama       # or openai, with a base url and keyticket-ai models                     # what that endpoint can reachticket-ai compose --title 'Etikettendruck bricht bei mehr als zehn Positionen ab'ticket-ai models --workflow          # an Actions workflow that drafts new issues

compose writes the body, measures it, hands the findings back to the model once, and prints the review to stderr so the body alone can be redirected. --fail-under makes it refuse to emit a draft that missed the house style.

Whichever model writes, the draft goes through the same measurement as any other ticket. That loop is why a small local model is usable here: it writes into a shape worked out by counting, and is marked against your team's own tickets afterwards.

A page instead

ticket-ai ui --lang de     # or en

A local page on 127.0.0.1:8760 with three tabs: the house style, a box to paste a draft into, and the open backlog worst-first. Loopback only, because this process holds a tracker token, and there is no flag to change that.

No build step and no CDN — one HTML file with its CSS and JavaScript inline.

Two languages, and they are separate

export TICKET_AI_UI_LANGUAGE=de       # buttons and headingsexport TICKET_AI_TICKET_LANGUAGE=de   # what it says to write tickets in

The distinction is easy to collapse and worth keeping. A German team may want the tool's own buttons in English; someone joining a German board still has to write the ticket in German.

TICKET_AI_TICKET_LANGUAGE overrides what the corpus measured and takes effect without re-learning. Leave it unset unless the board is mid-switch — a measurement beats a setting, and forcing a language the board does not use makes every existing ticket fail the language check.

Findings, the caveats under them and the list of what a ticket already got right are all rendered in the reader's language at the moment they are shown. The section names inside them are not: those are the team's own headings, and translating one turns it into a section the team does not have.

In CI

ticket-ai review "$CI_ISSUE" --fail-under 0.5

What it measures

The house style. Sections, length, labels, title markers, language — over a corpus it either took from you or found itself. Every finding cites a count over that corpus.

The tickets that failed, not only the ones that worked. A rate can be a comparison: not "78% of tickets have acceptance criteria" but "78% of the ones that shipped, and 30% of the ones that stalled". The second is evidence; the first invites a shrug.

The split is on outcome alone — a merged change, a reopen, a run of clarifying questions — and never on what the ticket contains, because splitting on content and then comparing content would be circular. Tickets a staleness bot closed are left out of both groups: those say something about attention rather than about writing. On a board where nothing separates the two, it says so, which is worth knowing before anyone is asked to write differently. docs/shipped-against-stalled.md has the guards and the real board that forced the bot exclusion.

Sections that only travel together. A section carried by 41% of tickets is under any threshold worth having — but if it is on 76% of the tickets that also have a Ziel section and 20% of the ones that do not, the team has a template and applies it to one kind of work. A board-wide rate hides that entirely.

The form you declared, against the tickets you got. With a .github/ISSUE_TEMPLATE or .gitlab/issue_templates in the checkout, gaps lines each field up against how often tickets actually carry it — and the gap runs both ways. A required field that turns up in 10% of tickets is a form asking for something people cannot easily give. A section most tickets carry that no form mentions is a convention the project grew and never wrote down.

What you need to write one. Give it a subject and it returns the related past tickets, the files the merge requests for those tickets actually changed, and the files in your checkout that mention it. That middle one lives only in the tracker's history — no amount of reading the code produces it, and it is usually the fastest way to find where the work will land.

How the mining works

Only closed tickets are sampled: an open ticket may be beautifully written, but nothing about it yet shows anyone could act on it. Each is scored on

Tickets opened and closed inside an hour are halved — usually duplicates or typo fixes, whose shape is not the shape of real work. Bot authors are dropped outright; learning a house style from Renovate is a real failure mode. No one author may supply more than 40% of the corpus, or the profile describes your most prolific ticket-writer instead of your team.

Every score carries the reasons that produced it. A corpus you cannot argue with is one you will not trust.

What it will not tell you

Whether your tickets are any good. Nothing here reads for meaning; it counts. It cannot tell you whether your acceptance criteria make sense. It can tell you that the 40 tickets that shipped in this project all had some and this one does not.

Alignment is not quality. The score is distance from the tickets that historically got built here. A one-line ticket from someone who knows exactly what they mean can score badly and be completely fine. The tool says this about itself, and so should you when you quote it at a colleague.

Nothing about your tracker's contents changes. Every operation is a read.

The ranking signals are also circumstantial. A well-written ticket closed as out-of-scope with no MR scores badly, and that is an acceptable error: the goal is thirty representative tickets, not the thirty best ones. A thin sample says so, loudly, in the report and in every review built on it.

Jira has no public API for linked branches and merge requests, so the "shipped" signal there falls back to remote links, and a board that does not post them cannot be split into shipped and stalled at all. It says that rather than reporting every ticket as stalled. That is a limit of the API, not of the corpus.

Measuring the part a model writes

Every test in this repository mocks the writer, so none of them says anything about the prose that comes back. evals/ is the answer to that: a frozen set of titles from public boards, each stored next to the board profile as it stood when the case was collected.

ticket-ai eval --baseline evals/baseline.json          # run it and mark the resultticket-ai eval --runs evals/results/latest.jsonl       # re-score records, no modelticket-ai eval --repeats 5 --markdown                  # spread, as a table for an MR

The cases were held out of the profile before it was built. A case the profile was fitted to flatters the model for a reason that has nothing to do with the model.

The runner records and does not score. A case costs thirty to seventy seconds on a local model, so scoring separately is what makes a changed metric free to try. --runs is that path.

Two of the figures are not in review_draft, deliberately. It asks whether the sections a board uses are present; it never asks what else the draft invented, because people rarely add a heading their board has never seen and models do it constantly. The other is spread: one number per case reads as precision a non-deterministic writer cannot support, so --repeats exists and the report prints the deviation next to the mean.

The judge, and why its number never travels alone

Counting cannot say whether the prose is about the title, and that is the failure worth catching: a draft can carry every section, hit the length, take the right labels and describe something else entirely. review_draft gives it full marks.

--judge asks a model that one question and nothing else. Not "is this a good ticket" - that is the question this tool exists to replace with counts.

ticket-ai eval --judge --labels evals/labels.jsonlpython tools/label_drafts.py evals/results/latest.jsonl   # collect the labels

Raw agreement is close to useless on its own. If nine drafts in ten are on topic, a judge that answers "on topic" every time agrees ninety percent of the time and has learned nothing. Cohen's kappa subtracts the agreement two people guessing at those rates would reach, so that judge scores zero.

The labels come from a person and there is no way around it. A calibration whose human half was generated compares one model to another. Without --labels, the judge line prints uncalibrated every time rather than documenting the caveat once and letting the number travel without it.

The dataset is 134 cases over ten public boards. Thirty are German, from kern-ux/pattern-library and fitko/fim/portal on gitlab.opencode.de, a public German GitLab that reads without a token. Three boards are Jira (KAFKA and CASSANDRA on issues.apache.org, HHH on hibernate.atlassian.net), and four write pure prose with no recurring sections at all.

That is not variety for its own sake. compose branches on each of them: a German prompt, a tracker abstraction, and a skeleton block that either lists sections or tells the model to write no headings at all. CI asserts all three survive, because a dataset that lost one would stop measuring it without the average moving, and the loss would be invisible in the one number people read.

kern-ux is why jsonl_lines exists: one of its tickets carries a literal U+2028, str.splitlines treats that as a line break and json.dumps does not escape it, so a valid file read as a truncated record and the error blamed the file.

evals/README.md has the rules about what may become a case. Public boards only.

Where it runs

The cheap half runs on every pull request that touches the harness: every board loads, no case is also an exemplar, and a committed results file is re-scored. No model, so it costs seconds.

The suite itself is workflow_dispatch. Composing 134 cases is minutes and money, and paying for it on a branch about a README typo buys nothing.

terraform/ is the report's address: an S3 bucket behind CloudFront, and an IAM role GitHub Actions assumes with a short-lived OIDC token. There is no AWS key in the repository secrets to leak or rotate. The trust policy is scoped to one repository; repo:owner/* would hand the role to every repository that owner ever creates. Everything works without any of it, and the publish step skips itself when the role variable is unset.

Documentation

Verified against

Run end to end, read-only, against home-assistant/core, pydantic/pydantic, astral-sh/uv, fastapi/fastapi (GitHub), inkscape/inkscape and gitlab-org/gitlab-runner (GitLab), hibernate.atlassian.net HHH (Jira Cloud), issues.apache.org/jira KAFKA (Jira Data Center), and one private GitLab board.

Home Assistant produces no conditional rules at all, which is the right answer: their issue form is mandatory, so every section already clears the board-wide threshold and a conditional has nothing to add.

Development

uv syncuv run pytestuv run ruff check .bash install_check.sh          # build the wheel and drive it from a clean venvuv run python tests/fleet.py   # the whole tool across 43 public boards

License

MIT.