Universal Test Framework

Universal Test Framework (UTF)

Contract-driven test enforcement and reporting for LLM-generated code — via MCP, VS Code, Copilot CLI, Claude, Cursor, or Python SDK.

UTF is an MCP server and contract enforcement engine. It does not replace your AI coding assistant — it gives every test your AI writes a mandatory quality gate, a structured audit trail, and a comprehensive compliance report.

[PyPI version] [Python] [License: MIT]

Why UTF? (Not Another Test Generator)

AI tools — GitHub Copilot, Claude, Cursor, Gemini — generate tests fast. The problem: fast ≠ trustworthy.

UTF vs Markdown Instructions / Prompt Files

You may already use markdown files (AGENTS.md, copilot-instructions.md, .cursorrules) to guide your AI. UTF is complementary — not competing:

Use markdown instructions to shape how your AI thinks. Use UTF to verify and report on what it produced.

Key Features

• 13 MCP tools — generate, validate, register, analyze, trace, execute, mutate, report, and query tests
• 6 languages — Python, TypeScript, JavaScript, Java, Go, C++
• 7 test types — unit, integration, API, E2E, contract, performance, security
• 8-section contract — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
• 3-segment test IDs — TC-{PRJ}-{MODULE}-{NNN} and classic TC-{MODULE}-{NNN} both accepted
• Five install modes — uvx (zero-clone), local clone, Docker, GitHub MCP Registry, pip SDK
• VS Code @utf — chat participant with slash commands
• CI/CD ready — GitHub Actions, GitLab CI, pre-commit hooks
• Per-project overrides — YAML rules in .utf/rules/ override global defaults

Quick Start

Setup Method Comparison

All methods (1, 2, 4) using stdio transport write the registry to ${workspaceFolder}/.utf/utf.db — isolated per project and persistent across sessions.

Method 1 — uvx (Zero-Install, Recommended)

Requires uv. No cloning, no venv.

Add to %APPDATA%\Code\User\mcp.json (Windows) or ~/.config/Code/User/mcp.json (macOS/Linux):

{  "servers": {    "utf": {      "type": "stdio",      "command": "uvx",      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],      "cwd": "${workspaceFolder}",      "env": {        "UTF_PROJECT_DIR": "${workspaceFolder}"      }    }  }}

Registry: .utf/utf.db inside ${workspaceFolder} — isolated per project, persistent across sessions.
Reload VS Code after editing mcp.json.

Method 2 — Local Clone (VS Code + Copilot)

git clone https://github.com/phoenice-labs/Universal-Test-Frameworkcd Universal-Test-Framework
# Register MCP server, install @utf extension, copy global prompts.\scripts\install-vscode-mcp.ps1
# Optionally scaffold a specific project.\scripts\install-vscode-mcp.ps1 -InitProject -ProjectDir C:\my-project

Reload VS Code → open Copilot Chat → ask naturally: generate e2e tests for my backend.

The install script writes this entry to mcp.json:

{  "utf": {    "type": "stdio",    "command": "<python>",    "args": ["-m", "mcp_server.server", "--transport", "stdio"],    "cwd": "${workspaceFolder}",    "env": { "PYTHONPATH": "<UTF_install_dir>" }  }}

Registry: ${workspaceFolder}/.utf/utf.db — isolated per project.
@utf slash commands are available after the VSIX extension is installed.

Method 3 — Docker (Team / Remote)

# Start the UTF servercd Universal-Test-Frameworkdocker compose -f docker/docker-compose.yml up -d
# MCP server available at http://localhost:8765/sse

Connect from VS Code by adding to mcp.json:

{  "servers": {    "utf-remote": {      "type": "sse",      "url": "http://localhost:8765/sse"    }  }}

Registry: persisted in a named Docker volume (utf_registry → /app/.utf/utf.db).
For per-project isolation with Docker, use a bind-mount in docker-compose.yml:

yaml

volumes:  - /path/to/your/project/.utf:/app/.utf

Because Docker uses HTTP/SSE transport (no ${workspaceFolder} templating), pass project_dir explicitly in tool calls, or set UTF_PROJECT_DIR in the container environment.

Per-project Docker workflow:

# docker-compose.override.ymlservices:  utf-server:    volumes:      - ./my-project/.utf:/app/.utf    environment:      UTF_PROJECT_DIR: /app

Method 4 — GitHub MCP Registry

Once published, UTF is discoverable via the MCP marketplace. Clients that support server.json install it automatically. The generated mcp.json entry is equivalent to Method 1 (uvx).

Registry isolation: The MCP registry schema does not support a cwd field at the registry level.
UTF resolves project isolation via (in priority order):

1. project_dir argument passed to each tool call
2. UTF_PROJECT_DIR environment variable
3. Path.cwd() fallback (server's working directory)

For correct isolation, ensure the MCP client writes "cwd": "${workspaceFolder}" and "UTF_PROJECT_DIR": "${workspaceFolder}" in the generated entry (UTF's mcp-gallery.json does this).

Method 5 — pip + Python SDK

pip install universal-test-framework

from mcp_server.tools.generate_tests import generate_testsfrom pathlib import Path
result = generate_tests(    test_type="unit",    source_code=open("src/auth.py").read(),    project_dir=str(Path.cwd()),   # ← pass explicitly for correct registry isolation)print(result["suite_code"])

Registry: <project_dir>/.utf/utf.db when project_dir is passed; falls back to Path.cwd().

MCP Tools Reference

The 8-Section Test Contract

Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are blocked — returned in blocked_tests, never silently included.

Minimum passing score: 0.85. Tests below this threshold are blocked regardless of individual section presence. "none" in gaps or vague rationale like "to test the function" are rejected.

Supported Matrix

The UTF 3-Phase E2E Workflow

This is the canonical flow for using UTF with any AI CLI (Copilot, Claude, Cursor) or VS Code. Follow the phases in order — skipping Phase 1 registration means the report has no Per-Test Contract Detail cards.

┌─────────────────────────────────────────────────────────────────────────┐│  PHASE 1 — CONTRACT GENERATION (LLM writes, UTF validates + registers) ││                                                                         ││  ① generate_tests (scaffold)                                           ││  ② LLM writes real test methods — each with its own TC-ID and         ││     8-section comment block (WHY / REQ / HOW / COV / OUT / GAP / MEAN)││  ③ validate_test_contract — score must be ≥ 0.85 per test             ││  ④ register_contracts — parse test file, upsert status=generated rows  ││  ⑤ generate_report — verify Per-Test Contract Detail cards appear      ││                                                                         ││  PHASE 2 — EXECUTION (pytest/jest/maven runs, results captured)        ││                                                                         ││  ⑥ pytest --junit-xml=utf-tests/reports/results.xml                   ││  ⑦ import_test_results — upsert executed/failed rows                   ││  ⑧ generate_report — now shows contract cards AND pass/fail status     ││                                                                         ││  PHASE 3 — HEALTH (ongoing coverage quality)                           ││                                                                         ││  ⑨ feedback_status — gap analysis, drift alerts, trend over 30 days   ││  ⑩ Address gaps → add tests → back to Phase 1                         │└─────────────────────────────────────────────────────────────────────────┘

Why register before running? The UTF registry has two record types. status=generated records (created by register_contracts) drive the Per-Test Contract Detail cards in the HTML report. status=executed/failed records (created by import_test_results) drive the Execution Results table. Both must exist for a test to appear in both sections. Running pytest before registering means you get execution rows but no 8-section detail cards.

Per-Method 8-Section Comment Block (mandatory)

Every test METHOD must have its own inline comment block — not a class docstring:

def test_health_returns_200(self, live_backend):    # ─── TC-FIQ-HLT-001 ──────────────────────────────────────────────────────    # WHY_GENERATED: The /health endpoint is the primary liveness signal for    #   load balancers and K8s probes. Non-200 = platform unavailable.    # REQUIREMENT_MAPPING: REQ-E2E-001    # HOW_IT_EXERCISES: GIVEN backend is running at http://localhost:8001    #   WHEN GET /health is called THEN HTTP 200 is returned.    # COVERAGE_CONTRIBUTION: Line coverage of health route; ~15% of health module    # EXPECTED_OUTCOME: HTTP 200; elapsed < 500ms    # GAPS_MISSING: Does not test health under load; no auth header tested    # MEANINGFULNESS_CHECK: Meaningful — gateway test for all other tests    # ─────────────────────────────────────────────────────────────────────────    r = requests.get(f"{live_backend}/health", timeout=5)    assert r.status_code == 200

Test ID formats accepted: TC-HLT-001 (2-segment) or TC-FIQ-HLT-001 (3-segment project-prefixed).

Invoking UTF from AI CLIs

GitHub Copilot CLI

UTF tools are called via natural language — no special syntax required:

# Phase 1 — Generate and registergenerate e2e tests for backend/app/api/routes/register contracts for utf-tests/test_myapp_e2e.pyutf report
# Phase 2 — After running pytestimport junit xml utf-tests/reports/results.xmlutf report
# Phase 3 — Health checkutf statuswhat are my coverage gaps?build a traceability matrix

Claude (claude.ai / Claude CLI / MCP client)

Claude supports MCP servers natively. With UTF added to your MCP config:

# Natural language triggers UTF MCP tools automatically"Generate e2e tests for my FastAPI backend at backend/app/""Register contracts for utf-tests/test_myapp_e2e.py""Generate the UTF report""Show UTF status and gaps""Validate this test against the 8-section contract: [paste test]"

To add UTF to Claude's MCP config (~/.config/claude/mcp.json or equivalent):

{  "mcpServers": {    "utf": {      "command": "uvx",      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],      "env": { "UTF_PROJECT_DIR": "/path/to/your/project" }    }  }}

Cursor

In Cursor, add UTF as an MCP server in .cursor/mcp.json (project-level) or ~/.cursor/mcp.json (global):

{  "mcpServers": {    "utf": {      "command": "uvx",      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],      "cwd": "${workspaceFolder}",      "env": { "UTF_PROJECT_DIR": "${workspaceFolder}" }    }  }}

Then ask Cursor naturally:

Generate unit tests for src/auth.py using UTFUTF reportRegister contracts for tests/test_api.py

Any MCP-Compatible Client (Windsurf, Continue, etc.)

The MCP entry is identical regardless of client:

{  "utf": {    "type": "stdio",    "command": "uvx",    "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],    "cwd": "${workspaceFolder}",    "env": { "UTF_PROJECT_DIR": "${workspaceFolder}" }  }}

UTF uses natural language detection — the same prompts work across all MCP-compatible AI clients.

GitHub Copilot CLI Usage

UTF is invoked via natural language in the GitHub Copilot CLI — there are no special slash commands or @utf syntax at the CLI prompt. Simply describe what you want and the MCP tools are called automatically.

How to Invoke UTF from the CLI

# In the GitHub Copilot CLI terminal (gh copilot / copilot-cli)generate unit tests for backend/app/routes/auth.pygenerate e2e tests covering REQ-001 through REQ-024register contracts for utf-tests/test_myapp_e2e.pyutf reportshow utf statusvalidate this test against the 8-section contractwhat are the coverage gaps?suggest test types for my projectbuild a traceability matrix

Full Command Reference (Natural Language → MCP Tool)

Registry Persistence

The UTF SQLite registry persists per-project across all sessions:

your-project/└── .utf/    ├── utf.db          ← SQLite registry (persists between sessions)    ├── utf-config.yaml ← Optional project overrides    ├── rules/          ← Optional YAML rule overrides    └── reports/        ├── contract_YYYYMMDD_HHMMSS.html        ├── contract_YYYYMMDD_HHMMSS.xml        └── contract_YYYYMMDD_HHMMSS.json

Once tests are generated (generate_tests), they are registered in .utf/utf.db. Subsequent calls to utf report, utf status, and query registry read from this persistent store — no re-generation required between sessions.

Providing Project Context

When the MCP server cannot infer your project root automatically, pass project_dir explicitly:

generate unit tests for src/auth.py in project C:/my-project

Or configure "cwd": "${workspaceFolder}" in your mcp.json (already set in the quickstart above) so the server always starts in the correct workspace.

VS Code Copilot Chat Integration

After running .\scripts\install-vscode-mcp.ps1 (Option 2) or adding the uvx MCP entry (Option 1):

Note: The @utf prefix and /slash-command syntax only work if you have the UTF VS Code Chat Participant extension installed (included via Option 2). In GitHub Copilot CLI and standard VS Code Copilot Chat without the extension, use natural language — the MCP tools are invoked automatically. See GitHub Copilot CLI Usage above.

VS Code Chat Participant Slash Commands (extension required)

Natural Language (no extension required)

In standard VS Code Copilot Chat or GitHub Copilot CLI, just ask:

generate unit tests for this fileshow utf statusutf reportwhat are my coverage gaps?validate this test against the contract

UTF reads the open file, detects language and framework, infers requirements from function signatures, generates happy-path + negative + edge-case tests — all validated against the 8-section contract.

Python SDK

Use UTF directly in scripts or CI pipelines without the MCP server:

from mcp_server.tools.generate_tests import generate_testsfrom mcp_server.tools.validate_contract import validate_test_contract
# Generate tests — UTF infers language, framework, and requirementsresult = generate_tests(    test_type="unit",    source_code=open("src/auth.py").read(),    requirements_text="US-101: Users must be authenticated before accessing dashboard",)
print(result["suite_code"])          # Executable test fileprint(result["traceability_matrix"]) # Requirements → tests mappingprint(result["gaps"])                # Identified coverage gaps
# Validate any existing testvalidation = validate_test_contract(test_content=my_test_markdown)print(f"Score: {validation['score']:.0%}  Valid: {validation['is_valid']}")

CI/CD Integration

GitHub Actions

# .github/workflows/test-quality-gate.ymlname: UTF Contract Gateon: [push, pull_request]jobs:  contract-gate:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - uses: actions/setup-python@v5        with: { python-version: "3.11" }      - run: pip install universal-test-framework      - name: Validate test contract        run: |          python - <<'EOF'          from mcp_server.tools.validate_contract import validate_test_contract          import glob, sys, pathlib
          failures = []          for f in glob.glob("tests/**/*.py", recursive=True):              content = pathlib.Path(f).read_text()              result = validate_test_contract(test_content=content)              if not result["is_valid"]:                  failures.append(f"{f}: score {result['score']:.0%}")          if failures:              print("Contract failures:\n" + "\n".join(failures))              sys.exit(1)          print("All tests passed contract validation")          EOF

Per-Project Configuration

Create .utf/rules/project-overrides.yaml in your project root:

# Raise minimum score for safety-critical codecontract:  min_score: 0.90          # default: 0.85  hard_block_below: 0.75
# Match your Jira project keytraceability:  requirement_id_pattern: "^(PROJ-\\d+|AC-\\d+(\\.\\d+)?|NFR-\\d+)$"
# Tests generated per requirement per typescenario_counts:  happy_path: 1  negative: 2  boundary: 1  edge_case: 1
# Coverage advisory thresholds (appear in report, do not block)coverage:  line_target: 85  branch_target: 75  mutation_score_target: 70

Rules are deep-merged: project overrides layer on top of UTF's global defaults. The 8-section contract structure itself cannot be overridden.

Architecture

UTF MCP Server (stdio or HTTP/SSE)│├── mcp_server/server.py         — FastMCP entrypoint, 13 registered tools│├── mcp_server/engine/           — Core processing│   ├── language_detector.py     — Detects language + framework from code/path│   ├── rule_engine.py           — Loads and merges YAML rules│   ├── contract_validator.py    — Enforces 8-section contract, scores tests│   ├── template_renderer.py     — Jinja2 test file generation│   ├── context_resolver.py      — Resolves project context for generation│   └── framework_mapper.py      — Maps language → framework → test runner│├── mcp_server/tools/            — One module per MCP tool├── mcp_server/registry/         — SQLite per-project test registry├── mcp_server/execution/        — pytest, Vitest, Maven, go-test adapters├── mcp_server/mutation/         — mutmut, Stryker, PIT, Gremlins adapters├── mcp_server/reporting/        — HTML, JUnit XML, JSON report generation├── mcp_server/feedback/         — CI listener, trend analysis, gap reopener│├── rules/                       — Global YAML rules (language, framework, type)├── templates/                   — Jinja2 test templates per language/framework├── agent-customization/         — VS Code copilot-instructions + prompt palette└── vscode-extension/            — @utf VS Code Chat Participant (VSIX)

Transport modes:

• stdio — default, used by VS Code MCP client and uvx
• HTTP/SSE — for remote/team deployment (--transport http --port 8765)

SQLite registry resolves to .utf/utf.db relative to the caller's project root (the cwd in mcp.json, or the project_dir parameter passed to any tool). Each project has its own isolated registry — no shared state. The registry persists across all sessions until explicitly cleared.

Installation Options Summary

Registry Isolation Rules

All setups resolve the SQLite registry path using the same priority chain:

1. project_dir argument (explicit per-tool call)2. UTF_PROJECT_DIR environment variable3. Path.cwd() at server start (fallback — avoid for multi-project use)

The recommended approach for all setups: set both "cwd": "${workspaceFolder}" and "env": { "UTF_PROJECT_DIR": "${workspaceFolder}" } in your mcp.json entry. This ensures registry isolation works even if a tool call omits project_dir.

Security

• All MCP tool inputs are validated via Pydantic before processing
• No secrets, credentials, or PII are logged or stored
• The registry (utf.db) is local to each project and never transmitted
• Docker image runs as non-root user
• Rate limiting is enforced on the HTTP/SSE transport

License

MIT — see LICENSE