Hotato

io.github.attenlabsPublished by GitHub account attenlabsv1.9.0Updated Oct 24, 2026

Open-source, self-hosted conversation QA for voice agents. MIT.

VerifiedSTDIODesktop onlyDeveloper ToolsSecurity & MonitoringAI & ML

Overview

AI-generated overview

Open-source, offline QA for voice-agent calls: scores recordings and traces for timing bugs and turns findings into CI failure checks.

What you can do
  • Score call audio and traces for say-do gaps, latency spikes, dead air and talk-over
  • Pull and check calls from voice platforms such as Vapi, Retell, Bland, Synthflow and Millis
  • Analyze local recordings and transcripts, writing HTML and JSON reports
  • Scan folders for trends and pin an incident into a portable, re-runnable failure check
  • Re-run stored checks to fail a build, and propose or verify findings through the MCP tools
When to use it
Useful when you build or operate a voice agent and want to catch regressions in turn timing, barge-in handling and latency, or gate CI on a specific recorded failure. It suits teams that prefer a local, offline tool over a hosted QA service.
Requirements
Runs locally over stdio as a Python package, launched with uvx from the hotato extra; Python and uv are needed. Platform checks read API keys from environment variables such as VAPI_API_KEY and RETELL_API_KEY. The core works offline with no account or key; optional self-hosted LLM scoring is a separate add-on.
Before you install
Platform commands send your API keys to those vendors' APIs and pull call data onto your machine. Pinning and proving checks writes reports and evidence databases to disk, and the exit code gates builds. Timing findings need two-channel audio; mono or mixed exports are refused as not scorable.

Installation

In SourceWeft

  1. Open Hotato in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

[hotato]

[PyPI version] [Downloads per month] [Python versions] [CI status] [MIT license]

hotato

Find what broke in your agent calls. Pin it and CI stays red until you fix it.

bash
pip install hotatohotato check --demo             # a bundled failing call: what broke, and whenhotato check ./call.wav         # then your own recordinghotato vapi health              # or your last 100 Vapi calls

Zero config. Vapi, Retell, Bland, Synthflow, Millis, or local audio. Works with any recording: mono, dual-channel, or a transcript. Timing math, not a judge. Offline. Free. MIT.

hotato.dev

What it finds

  • Say-do gaps: the caller interrupts to cancel (a barge-in), the agent says "canceled", the booking tool fires anyway. hotato takes the turn timing from the audio and the tool call from your OTel trace.
  • Latency spikes: the pause before a reply going from 800 ms to over 2 s.
  • Dead air: that pause reaching 5 s, or the line going quiet.
  • Talk-over: the agent starts a fresh utterance over the caller.

Quickstart

Vapi

bash
pip install hotatoexport VAPI_API_KEY=...hotato vapi health --last 7d --output report.html

Open report.html: every critical incident, timestamped, and your Voice Stability Score.

Retell

bash
export RETELL_API_KEY=...hotato retell health --call-id CALL_ID

--call-id is required and repeatable: you name the Retell calls to pull. hotato bland health, hotato synthflow health, and hotato millis health follow the Vapi shape. Those stacks mix both voices onto one channel, so they measure silence timing, dead air and latency gaps, each finding with its measured confidence; barge-in and talk-over need two channels.

Local audio

bash
hotato autopsy ./call.wav

Writes a self-contained HTML report to hotato-output/, plus the JSON pin reads. Open the HTML in your browser.

From finding a bug to gating on it

autopsy turns one recording into timestamped incidents. scan reads a folder and tracks the trend. When you are ready, move a finding into CI: hotato pin turns one incident into a portable failure check, and hotato prove re-runs every stored check and fails the build rather than pass on evidence it cannot re-read. Every verdict carries its own evidence across five dimensions: outcome, policy, conversation, speech, reliability.

A pinned check re-measures its own stored recording, so it holds the build red on that bug and catches an edited bundle, a loosened policy, or an engine upgrade that scores the same bytes differently. Clearing it takes a fresh recording of the same moment against the current agent: hotato drive <bundle> places that call on Vapi or Twilio, and docs/RECAPTURE.md is the walkthrough for every other stack.

Pin a bug →

For continuous use: run hotato vapi health on a schedule, and open hotato console --production-db evidence.db to watch calls land live.

Wire it into CI

The exit code is the verdict: 0 pass, 1 fail, 2 refuse (could not tell).

yaml
# .github/workflows/voice-qa.ymlon: [pull_request]jobs:  hotato:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - uses: attenlabs/[email protected]        with:          contracts: contracts/          hotato-version: 1.20.0

contracts/ holds what pin wrote. Full workflow: docs/CI.md.

Point your agent at it

Point Claude Code, Cursor, or any coding agent at this repo: it reads AGENTS.md and runs the loop end to end offline, no key. Over local stdio the MCP server adds the scorer plus read/verify/propose tools: uvx --from "hotato[mcp]" hotato-mcp (docs/MCP.md). Deploying is yours.

Nothing leaves your machine

hotato runs offline, on the machine that invokes it. The core is stdlib-only Python: no account, no key, no network call of its own. Your traces, prompts, and audio stay on your disk. Scoring with a language model you host yourself is a separate opt-in add-on, outside that core.

Go deeper

The whole loop, command by command: docs/LIFECYCLE.md. First touch to a CI gate: docs/GETTING-STARTED.md. Feed it what you already have: docs/CONNECT.md · docs/TRACE.md · docs/SIMULATE.md. What every verdict stands on: docs/EVIDENCE-CONTRACT.md. Next to the hosted alternatives: docs/COMPARE.md.

The deep toolkit -- capture, simulation, load, benchmarking, the fix ladder, the fleet control plane -- lives under hotato lab (hotato lab --help). The public commands are durable. hotato lab moves faster, and every command name that worked before 1.17 still runs unchanged.

Specifications

PropertyValue
Footprint~10 MiB installed, 0 runtime dependencies (stdlib-only)
Reproducibilitybyte-for-byte: the same recording, the same report
Exit codes0 pass · 1 fail · 2 refuse
Release integrityOIDC Trusted Publishing + build-provenance attested
Runtimeoffline, off the production data path
Verify the measurement yourself
bash
PYTHONPATH=src python3 -m hotato.benchmark \  --scenarios corpus/real/scenarios --audio corpus/real/audio

On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is 20 ms. Provenance: corpus/real/README.md · method: METHODOLOGY.md.

Timing is measurable only when the two voices arrive on separate channels; a mono or mixed export is marked NOT SCORABLE and refused (hotato trust --stereo call.wav). The full four-tier evidence policy (what each verdict stands on, per input) is docs/EVIDENCE-CONTRACT.md.

Contribute

Issues and PRs welcome: CONTRIBUTING.md · SECURITY.md · CHANGELOG · docs/

License

MIT (LICENSE)

Know when to pass it on.

mcp-name: io.github.attenlabs/hotato

Source: README.md at commit f54713a

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v1.9.0LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts