Judges Panel

Judges Panel

An MCP (Model Context Protocol) server that provides a panel of 45 specialized judges to evaluate AI-generated code — acting as an independent quality gate regardless of which project is being reviewed. Combines deterministic pattern matching & AST analysis (instant, offline, zero LLM calls) with LLM-powered deep-review prompts that let your AI assistant perform expert-persona analysis across all 45 domains.

Highlights:

• Includes an App Builder Workflow (3-step) demo for release decisions, plain-language risk summaries, and prioritized fixes — see Try the Demo.
• Includes V2 context-aware evaluation with policy profiles, evidence calibration, specialty feedback, confidence scoring, and uncertainty reporting.
• Includes public repository URL reporting to clone a repo, run the full tribunal, and output a consolidated markdown report.
• 200+ deterministic auto-fix patches (see src/patches/index.ts) plus LLM-powered deep review.

🧪 Many commands in printHelp are experimental/roadmap. By default, we show GA commands only. Set JUDGES_SHOW_EXPERIMENTAL=1 to reveal stubs; these may not be wired yet.

[CI] [npm] [npm downloads] [License: MIT] [Tests]

🔰 Packages

• CLI: @kevinrabun/judges-cli → binary judges (use npx @kevinrabun/judges-cli eval --file app.ts).
• MCP/API: @kevinrabun/judges → programmatic API + MCP server (npm install @kevinrabun/judges).
• VS Code extension: see vscode-extension/.
• GitHub Action: uses: KevinRabun/judges@main (see CI quickstart).

Quickstart

CLI (one-off)

# Using the CLI package (recommended)npx @kevinrabun/judges-cli eval --file src/app.ts
# Show GA commands only (default)npx @kevinrabun/judges-cli --help
# Show experimental/roadmap commandsecho "JUDGES_SHOW_EXPERIMENTAL=1" >> $GITHUB_ENVnpx @kevinrabun/judges-cli --help
# License scan (supply-chain & license compliance)npx @kevinrabun/judges-cli license-scan --dir .

CLI vs API: If you want to embed Judges in your app (MCP/API), install @kevinrabun/judges. For the command-line, use @kevinrabun/judges-cli (binary judges).

GitHub Action

name: Judgeson: [pull_request, push]jobs:  judges:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - uses: KevinRabun/judges@main        with:          path: .          diff-only: true           # evaluate only changed lines in PRs (default true)          fail-on-findings: true    # fail on critical/high findings          upload-sarif: true        # upload SARIF to GitHub Code Scanning

Programmatic API (MCP server included)

npm install @kevinrabun/judges

import { evaluateCode } from "@kevinrabun/judges/api";const verdict = evaluateCode("const password = 'ProdSecret';", "typescript");console.log(verdict.overallVerdict, verdict.overallScore);

MCP server

The MCP server runs on stdio and is started by your MCP client (VS Code, Claude Desktop, etc.). Configure it in your MCP settings (e.g. mcp.json):

{  "servers": {    "judges": {      "type": "stdio",      "command": "npx",      "args": ["-y", "@kevinrabun/judges"]    }  }}

Or run the server directly:

npx @kevinrabun/judges# Starts the MCP server on stdio

Config file: .judgesrc.json (supports ${ENV_VAR} substitution via expandEnvPlaceholders). See Configuration.

Why Judges?

AI code generators (Copilot, Cursor, Claude, ChatGPT, etc.) write code fast — but they routinely produce insecure defaults, missing auth, hardcoded secrets, and poor error handling. Human reviewers catch some of this, but nobody reviews 45 dimensions consistently.

Judges doesn't replace linters — it covers the dimensions linters don't: authentication strategy, data sovereignty, cost patterns, accessibility, framework-specific anti-patterns, and architectural issues across multiple files.

[Judges — Terminal Output]

Quick Start

Prereqs: Node.js >=18 (>=20 recommended), npx available. The judges CLI binary ships with @kevinrabun/judges-cli (preferred) and also works via npx @kevinrabun/judges.

Packages:

• CLI: npm install -g @kevinrabun/judges-cli (or npx @kevinrabun/judges-cli ...)
• MCP/API: npm install @kevinrabun/judges

Use @kevinrabun/judges for the MCP server and programmatic API. Use @kevinrabun/judges-cli when you want the judges terminal command.

Try it now (no clone needed)

# Install the CLI globallynpm install -g @kevinrabun/judges-cli
# Evaluate any filejudges eval src/app.ts
# Pipe from stdincat api.py | judges eval --language python
# Single judgejudges eval --judge cybersecurity server.ts
# SARIF output for CIjudges eval --file app.ts --format sarif > results.sarif
# HTML report with severity filters and dark/light themejudges eval --file app.ts --format html > report.html
# Fail CI on findings (exit code 1)judges eval --fail-on-findings src/api.ts
# Suppress known findings via baselinejudges eval --baseline baseline.json src/api.ts
# Use a named presetjudges eval --preset security-only src/api.ts
# Use a config filejudges eval --config .judgesrc.json src/api.ts
# Set a minimum score threshold (exit 1 if below)judges eval --min-score 80 src/api.ts
# One-line summary for scriptsjudges eval --summary src/api.ts
# Agentic skills (orchestrated judge sets)judges skill ai-code-review --file src/app.tsjudges skill security-review --file src/api.ts --format jsonjudges skill release-gate --file src/app.tsjudges skills   # list available skills
> Full catalog: [`docs/skills.md`](docs/skills.md)

# List all 45 judgesjudges list

Additional CLI Commands

# Interactive project setup wizardjudges init
# Preview auto-fix patches (dry run)judges fix src/app.ts
# Apply patches directlyjudges fix src/app.ts --apply
# License compliance scan (copyleft/unknown detection)judges license-scan --format json --risk high
# Watch mode — re-evaluate on file savejudges watch src/
# Project-level report (local directory)judges report . --format html --output report.html
# Evaluate a unified diff (pipe from git diff)git diff HEAD~1 | judges diff
# Analyze dependencies for supply-chain risksjudges deps --path . --format json
# Run GitHub App server (zero-config PR reviews)judges app serve --port 4567
# Run GitHub PR review (gh CLI required)judges review --pr 123 --repo owner/name --diff-only
# Auto-tune presets and configsjudges tune --dir . --apply
# Create a baseline file to suppress known findingsjudges baseline create --file src/api.ts -o baseline.json
# Generate CI template filesjudges ci-templates --provider githubjudges ci-templates --provider gitlabjudges ci-templates --provider azurejudges ci-templates --provider bitbucket
# Generate per-judge rule documentationjudges docsjudges docs --judge cybersecurityjudges docs --output docs/
# Install shell completionsjudges completions bash   # eval "$(judges completions bash)"judges completions zshjudges completions fishjudges completions powershell
# Install pre-commit hookjudges hook install
# Uninstall pre-commit hookjudges hook uninstall

🔎 Tip: The CLI help now defaults to GA commands only. To see experimental/roadmap commands, run:

bash

JUDGES_SHOW_EXPERIMENTAL=1 judges --help

GitHub App (self-hosted webhook)

Run a zero-config PR reviewer as a GitHub App:

# Run the webhook server locallyjudges app serve --port 4567

Required env vars:

• JUDGES_APP_ID – GitHub App ID
• JUDGES_PRIVATE_KEY or JUDGES_PRIVATE_KEY_PATH – PEM private key
• JUDGES_WEBHOOK_SECRET – signature verification secret

Optional:

• JUDGES_MIN_SEVERITY (default: medium)
• JUDGES_MAX_COMMENTS (default: 25)
• JUDGES_TEST_DRY_RUN=1 to avoid live network calls during tests

For local testing, you can expose http://localhost:4567/webhook via ngrok http 4567 and configure the GitHub App webhook URL accordingly.

Use in GitHub Actions

Add Judges to your CI pipeline with zero configuration:

# .github/workflows/judges.ymlname: Judges Code Reviewon: [pull_request]
jobs:  judges:    runs-on: ubuntu-latest    permissions:      contents: read      security-events: write  # only if using upload-sarif    steps:      - uses: actions/checkout@v4      - uses: KevinRabun/judges@main        with:          path: src/api.ts        # file or directory          format: text             # text | json | sarif | markdown          upload-sarif: true       # upload to GitHub Code Scanning          fail-on-findings: true   # fail CI on critical/high findings

Outputs available for downstream steps: verdict, score, findings, critical, high, sarif-file.

Use with Docker (no Node.js required)

# Build the imagedocker build -t judges .
# Evaluate a local filedocker run --rm -v $(pwd):/code judges eval --file /code/app.ts
# Pipe from stdincat api.py | docker run --rm -i judges eval --language python
# List judgesdocker run --rm judges list

Or use as an MCP server

1. Install and Build

git clone https://github.com/KevinRabun/judges.gitcd judgesnpm installnpm run build

2. Try the Demo

Run the included demo to see all 45 judges evaluate a purposely flawed API server:

npm run demo

This evaluates examples/sample-vulnerable-api.ts — a file intentionally packed with security holes, performance anti-patterns, and code quality issues — and prints a full verdict with per-judge scores and findings.

The demo now also includes an App Builder Workflow (3-step) section. In a single run, you get both tribunal output and workflow output:

• Release decision (Ship now / Ship with caution / Do not ship)
• Plain-language summaries of top risks
• Prioritized remediation tasks and AI-fixable P0/P1 items

Sample workflow output (truncated):

╔══════════════════════════════════════════════════════════════╗║             App Builder Workflow Demo (3-Step)             ║╚══════════════════════════════════════════════════════════════╝
  Decision       : Do not ship  Verdict        : FAIL (47/100)  Risk Counts    : Critical 24 | High 27 | Medium 55
  Step 2 — Plain-Language Findings:  - [CRITICAL] DATA-001: Hardcoded password detected      What: ...      Why : ...      Next: ...
  Step 3 — Prioritized Tasks:  - P0 | DEVELOPER | Effort L | DATA-001      Task: ...      Done: ...
  AI-Fixable Now (P0/P1):  - P0 DATA-001: ...

Sample tribunal output (truncated):

╔══════════════════════════════════════════════════════════════╗║           Judges Panel — Full Tribunal Demo                 ║╚══════════════════════════════════════════════════════════════╝
  Overall Verdict : FAIL  Overall Score   : 43/100  Critical Issues : 15  High Issues     : 17  Total Findings  : 83  Judges Run      : 33
  Per-Judge Breakdown:  ────────────────────────────────────────────────────────────────  ❌ Judge Data Security              0/100    7 finding(s)  ❌ Judge Cybersecurity              0/100    7 finding(s)  ❌ Judge Cost Effectiveness        52/100    5 finding(s)  ⚠️  Judge Scalability              65/100    4 finding(s)  ❌ Judge Cloud Readiness           61/100    4 finding(s)  ❌ Judge Software Practices        45/100    6 finding(s)  ❌ Judge Accessibility              0/100    8 finding(s)  ❌ Judge API Design                 0/100    9 finding(s)  ❌ Judge Reliability               54/100    3 finding(s)  ❌ Judge Observability             45/100    5 finding(s)  ❌ Judge Performance               27/100    5 finding(s)  ❌ Judge Compliance                 0/100    4 finding(s)  ⚠️  Judge Testing                  90/100    1 finding(s)  ⚠️  Judge Documentation            70/100    4 finding(s)  ⚠️  Judge Internationalization     65/100    4 finding(s)  ⚠️  Judge Dependency Health        90/100    1 finding(s)  ❌ Judge Concurrency               44/100    4 finding(s)  ❌ Judge Ethics & Bias             65/100    2 finding(s)  ❌ Judge Maintainability           52/100    4 finding(s)  ❌ Judge Error Handling            27/100    3 finding(s)  ❌ Judge Authentication             0/100    4 finding(s)  ❌ Judge Database                   0/100    5 finding(s)  ❌ Judge Caching                   62/100    3 finding(s)  ❌ Judge Configuration Mgmt         0/100    3 finding(s)  ⚠️  Judge Backwards Compat         80/100    2 finding(s)  ⚠️  Judge Portability              72/100    2 finding(s)  ❌ Judge UX                        52/100    4 finding(s)  ❌ Judge Logging Privacy            0/100    4 finding(s)  ❌ Judge Rate Limiting             27/100    4 finding(s)  ⚠️  Judge CI/CD                    80/100    2 finding(s)

3. Run the Tests

npm test

Runs automated tests covering all judges, AST parsers, markdown formatters, and edge cases.

4. Connect to Your Editor

VS Code (recommended — zero config)

Install the Judges Panel extension from the Marketplace. It provides:

• Inline diagnostics & quick-fixes on every file save
• @judges chat participant — type @judges in Copilot Chat, or just ask for a "judges panel review" and Copilot routes automatically
• Auto-configured MCP server — all 45 expert-persona prompts available to Copilot with zero setup

code --install-extension kevinrabun.judges-panel

VS Code — manual MCP config

If you prefer explicit workspace config (or want teammates without the extension to benefit), create .vscode/mcp.json:

{  "servers": {    "judges": {      "command": "npx",      "args": ["-y", "@kevinrabun/judges"]    }  }}

Claude Desktop

Add to claude_desktop_config.json:

{  "mcpServers": {    "judges": {      "command": "npx",      "args": ["-y", "@kevinrabun/judges"]    }  }}

Cursor / other MCP clients

Use the same npx command for any MCP-compatible client:

{  "command": "npx",  "args": ["-y", "@kevinrabun/judges"]}

5. Use Judges in GitHub Copilot PR Reviews

Yes — users can include Judges as part of GitHub-based review workflows, with one important caveat:

• The hosted copilot-pull-request-reviewer on GitHub does not currently let you directly attach arbitrary local MCP servers the same way VS Code does.
• The practical pattern is to run Judges in CI on each PR, publish a report/check, and have Copilot + human reviewers use that output during review.

Option A (recommended): PR workflow check + report artifact

Create .github/workflows/judges-pr-review.yml:

name: Judges PR Review
on:  pull_request:    types: [opened, synchronize, reopened]
jobs:  judges:    runs-on: ubuntu-latest    permissions:      contents: read      pull-requests: write
    steps:      - name: Checkout        uses: actions/checkout@v4
      - name: Setup Node        uses: actions/setup-node@v4        with:          node-version: 20          cache: npm
      - name: Install        run: npm ci
      - name: Generate Judges report        run: |          npx tsx -e "import { generateRepoReportFromLocalPath } from './src/reports/public-repo-report.ts';          const result = generateRepoReportFromLocalPath({            repoPath: process.cwd(),            outputPath: 'judges-pr-report.md',            maxFiles: 600,            maxFindingsInReport: 150,          });          console.log('Overall:', result.overallVerdict, result.averageScore);"
      - name: Upload report artifact        uses: actions/upload-artifact@v4        with:          name: judges-pr-report          path: judges-pr-report.md

This gives every PR a reproducible Judges output your team (and Copilot) can reference.

Option B: Add Copilot custom instructions in-repo

Add .github/instructions/judges.instructions.md with guidance such as:

When reviewing pull requests:1. Read the latest Judges report artifact/check output first.2. Prioritize CRITICAL and HIGH findings in remediation guidance.3. If findings conflict, defer to security/compliance-related Judges.4. Include rule IDs (e.g., DATA-001, CYBER-004) in suggested fixes.

This helps keep Copilot feedback aligned with Judges findings.

CLI Reference

All commands support --help for usage details.

judges eval

Evaluate a file with all 45 judges or a single judge.

judges init

Interactive wizard that generates project configuration:

• .judgesrc.json — rule customization, disabled judges, severity thresholds
• .github/workflows/judges.yml — GitHub Actions CI workflow
• .gitlab-ci.judges.yml — GitLab CI pipeline (optional)
• azure-pipelines.judges.yml — Azure Pipelines (optional)

judges fix

Preview or apply auto-fix patches from deterministic findings.

judges watch

Continuously re-evaluate files on save.

judges report

Run a full project-level tribunal on a local directory.

judges hook

Manage a Git pre-commit hook that runs Judges on staged files.

judges hook install    # add pre-commit hookjudges hook uninstall  # remove pre-commit hook

Detects Husky (.husky/pre-commit) and falls back to .git/hooks/pre-commit. Uses marker-based injection so it won't clobber existing hooks.

judges diff

Evaluate only the changed lines from a unified diff (e.g., git diff output).

git diff HEAD~1 | judges diffjudges diff --file changes.patch --format sarif

judges deps

Analyze project dependencies for supply-chain risks.

judges deps --path .judges deps --path ./backend --format json

judges baseline

Create a baseline file to suppress known findings in future evaluations.

judges baseline create --file src/api.tsjudges baseline create --file src/api.ts -o .judges-baseline.json

judges ci-templates

Generate CI/CD configuration templates for popular providers.

judges ci-templates --provider github   # .github/workflows/judges.ymljudges ci-templates --provider gitlab   # .gitlab-ci.judges.ymljudges ci-templates --provider azure    # azure-pipelines.judges.ymljudges ci-templates --provider bitbucket # bitbucket-pipelines.yml (snippet)

judges docs

Generate per-judge rule documentation in Markdown.

judges docs                          # all judges to stdoutjudges docs --judge cybersecurity    # single judgejudges docs --output docs/judges/    # write files to directory

judges completions

Generate shell completion scripts.

eval "$(judges completions bash)"        # Basheval "$(judges completions zsh)"         # Zshjudges completions fish | source         # Fishjudges completions powershell            # PowerShell (Register-ArgumentCompleter)

Named Presets

Use --preset to apply pre-configured evaluation settings:

judges eval --preset security-only src/api.tsjudges eval --preset strict --format sarif src/app.ts > results.sarif

CI Output Formats

JUnit XML

Generate JUnit XML for Jenkins, Azure DevOps, GitHub Actions, or GitLab test result viewers:

judges eval --format junit src/api.ts > results.xml

Each judge maps to a <testsuite>, each finding becomes a <testcase> with <failure> for critical/high severity.

CodeClimate / GitLab Code Quality

Generate CodeClimate JSON for GitLab Code Quality or similar tools:

judges eval --format codeclimate src/api.ts > codequality.json

Score Badges

Generate SVG or text badges for your README:

import { generateBadgeSvg, generateBadgeText } from "@kevinrabun/judges/badge";
const svg = generateBadgeSvg(85);          // shields.io-style SVGconst text = generateBadgeText(85);        // "✓ judges 85/100"const svg2 = generateBadgeSvg(75, "quality"); // custom label

The Judge Panel

How It Works

The tribunal operates in three layers:

1. Pattern-Based Analysis — All tools (evaluate_code, evaluate_code_single_judge, evaluate_project, evaluate_diff) perform heuristic analysis using regex pattern matching to catch common anti-patterns. This layer is instant, deterministic, and runs entirely offline with zero external API calls.

2. AST-Based Structural Analysis — The Code Structure judge (STRUCT-* rules) uses real Abstract Syntax Tree parsing to measure cyclomatic complexity, nesting depth, function length, parameter count, dead code, and type safety with precision that regex cannot achieve. All supported languages — TypeScript, JavaScript, Python, Rust, Go, Java, C#, and C++ — are parsed via tree-sitter WASM grammars (real syntax trees compiled to WebAssembly, in-process, zero native dependencies). A scope-tracking structural parser is kept as a fallback when WASM grammars are unavailable. No external AST server required.

3. LLM-Powered Deep Analysis (Prompts) — The server exposes MCP prompts (e.g., judge-data-security, judge-cybersecurity) that provide each judge's expert persona as a system prompt. When used by an LLM-based client (Copilot, Claude, Cursor, etc.), the host LLM performs deeper, context-aware probabilistic analysis beyond what static patterns can detect. This is where the systemPrompt on each judge comes alive — Judges itself makes no LLM calls, but it provides the expert criteria so your AI assistant can act as 45 specialized reviewers.

Composable by Design

Judges Panel is a dual-layer review system: instant deterministic tools (offline, no API keys) for pattern and AST analysis, plus 45 expert-persona MCP prompts that unlock LLM-powered deep analysis when connected to an AI client. It does not try to be a CVE scanner or a linter. Those capabilities belong in dedicated MCP servers that an AI agent can orchestrate alongside Judges.

Built-in AST Analysis

Unlike earlier versions that recommended a separate AST MCP server, Judges Panel now includes real AST-based structural analysis out of the box:

• TypeScript, JavaScript, Python, Rust, Go, Java, C#, C++ — All parsed with a unified tree-sitter WASM engine for full syntax-tree analysis (functions, complexity, nesting, dead code, type safety). Falls back to a scope-tracking structural parser when WASM grammars are unavailable

The Code Structure judge (STRUCT-*) uses these parsers to accurately measure:

Recommended MCP Stack

When your AI coding assistant connects to multiple MCP servers, each one contributes its specialty:

┌─────────────────────────────────────────────────────────┐│                   AI Coding Assistant                   ││              (Claude, Copilot, Cursor, etc.)            │└──────┬──────────────────┬──────────┬───────────────────┘       │                  │          │       ▼                  ▼          ▼  ┌──────────────┐  ┌────────┐  ┌────────┐  │   Judges     │  │  CVE / │  │ Linter │  │   Panel      │  │  SBOM  │  │ Server │  │ ─────────────│  └────────┘  └────────┘  │ 44 Heuristic │   Vuln DB     Style &  │   judges     │   scanning    correctness  │ + AST judge  │  └──────────────┘   Patterns +   structural   analysis

What This Means in Practice

When you ask your AI assistant "Is this code production-ready?", the agent can:

1. Judges Panel → Scan for hardcoded secrets, missing error handling, N+1 queries, accessibility gaps, compliance issues, plus analyze cyclomatic complexity, detect dead code, and flag deeply nested functions via AST
2. CVE Server → Check every dependency in package.json against known vulnerabilities
3. Linter Server → Enforce team style rules, catch language-specific gotchas

Each server returns structured findings. The AI synthesizes everything into a single, actionable review — no single server needs to do it all.

MCP Tools

evaluate_v2

Run a V2 context-aware tribunal evaluation designed to raise feedback quality toward lead engineer/architect-level review:

• Policy profile calibration (default, startup, regulated, healthcare, fintech, public-sector)
• Context ingestion (architecture notes, constraints, standards, known risks, data-boundary model)
• Runtime evidence hooks (tests, coverage, latency, error rate, vulnerability counts)
• Specialty feedback aggregation by judge/domain
• Confidence scoring and explicit uncertainty reporting

Supports:

• Code mode: code + language
• Project mode: files[]

evaluate_app_builder_flow

Run a 3-step app-builder workflow for technical and non-technical stakeholders:

1. Tribunal review (code/project/diff)
2. Plain-language translation of top risks
3. Prioritized remediation tasks with AI-fixable P0/P1 extraction

Supports:

• Code mode: code + language
• Project mode: files[]
• Diff mode: code + language + changedLines[]

evaluate_public_repo_report

Clone a public repository URL, run the full judges panel across eligible source files, and generate a consolidated markdown report.

Quick examples

Generate a report from CLI:

npm run report:public-repo -- --repoUrl https://github.com/microsoft/vscode --output reports/vscode-judges-report.md
# stricter credential-signal mode (optional)npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --credentialMode strict --output reports/openclaw-judges-report-strict.md
# judge findings only (exclude AST/code-structure findings)npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --includeAstFindings false --output reports/openclaw-judges-report-no-ast.md
# show only findings at 80%+ confidencenpm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --minConfidence 0.8 --output reports/openclaw-judges-report-high-confidence.md
# include must-fix gate summary in the generated reportnpm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --enableMustFixGate true --mustFixMinConfidence 0.9 --mustFixDangerousPrefix AUTH --mustFixDangerousPrefix CYBER --output reports/openclaw-judges-report-mustfix.md
# opinionated quick-start mode (recommended first run)npm run report:quickstart -- --repoUrl https://github.com/openclaw/openclaw --output reports/openclaw-quickstart.md

Call from MCP client:

{  "tool": "evaluate_public_repo_report",  "arguments": {    "repoUrl": "https://github.com/microsoft/vscode",    "branch": "main",    "maxFiles": 400,    "maxFindingsInReport": 120,    "credentialMode": "strict",    "includeAstFindings": false,    "minConfidence": 0.8,    "enableMustFixGate": true,    "mustFixMinConfidence": 0.9,    "mustFixDangerousRulePrefixes": ["AUTH", "CYBER", "DATA"],    "outputPath": "reports/vscode-judges-report.md"  }}

Typical response summary includes:

• overall verdict and average score
• analyzed file count and total findings
• per-judge score table
• highest-risk findings and lowest-scoring files

Sample report snippet:

# Public Repository Full Judges Report
Generated from https://github.com/microsoft/vscode on 2026-02-21T12:00:00.000Z.
## Executive Summary- Overall verdict: WARNING- Average file score: 78/100- Total findings: 412 (critical 3, high 29, medium 114, low 185, info 81)

get_judges

List all available judges with their domains and descriptions.

evaluate_code

Submit code to the full judges panel. all 45 judges evaluate independently and return a combined verdict.

evaluate_code_single_judge

Submit code to a specific judge for targeted review.

evaluate_project

Submit multiple files for project-level analysis. all 45 judges evaluate each file, plus cross-file architectural analysis detects code duplication, inconsistent error handling, and dependency cycles.

evaluate_diff

Evaluate only the changed lines in a code diff. Runs all 45 judges on the full file but filters findings to lines you specify. Ideal for PR reviews and incremental analysis.

analyze_dependencies

Analyze a dependency manifest file for supply-chain risks, version pinning issues, typosquatting indicators, and dependency hygiene. Supports package.json, requirements.txt, Cargo.toml, go.mod, pom.xml, and .csproj files.

evaluate_git_diff

Evaluate only changed lines from a git diff. Provide either repoPath for a live git diff or diffText for a pre-computed unified diff.

re_evaluate_with_context

Re-run the tribunal with prior findings as context for iterative refinement. Supports dispute resolution, developer context injection, and focus-area filtering.

Additional MCP Tools

Judge IDs

data-security · cybersecurity · security · cost-effectiveness · scalability · cloud-readiness · software-practices · accessibility · api-design · api-contract · reliability · observability · performance · compliance · data-sovereignty · testing · documentation · internationalization · dependency-health · concurrency · ethics-bias · maintainability · error-handling · authentication · database · caching · configuration-management · backwards-compatibility · portability · ux · logging-privacy · rate-limiting · ci-cd · code-structure · agent-instructions · ai-code-safety · framework-safety · iac-security · hallucination-detection · intent-alignment · multi-turn-coherence · model-fingerprint · over-engineering · logic-review · false-positive-review

MCP Prompts

Each judge has a corresponding prompt for LLM-powered deep analysis:

Configuration

Create a .judgesrc.json (or .judgesrc) file in your project root to customize evaluation behavior. See .judgesrc.example.json for a copy-paste-ready template, or reference the JSON Schema for full IDE autocompletion.

{  "$schema": "https://github.com/KevinRabun/judges/blob/main/judgesrc.schema.json",  "preset": "strict",  "minSeverity": "medium",  "disabledRules": ["COST-*", "I18N-001"],  "disabledJudges": ["accessibility", "ethics-bias"],  "ruleOverrides": {    "SEC-003": { "severity": "critical" },    "DOC-*": { "disabled": true }  },  "languages": ["typescript", "python"],  "format": "text",  "failOnFindings": false,  "baseline": "",  "regulatoryScope": ["GDPR", "PCI-DSS", "SOC2"],  "consensusThreshold": 0.7}

All evaluation tools (CLI and MCP) accept the same configuration fields via --config <path> or inline config parameter.

Advanced Features

Inline Suppressions

Suppress specific findings directly in source code using comment directives:

const x = eval(input); // judges-ignore SEC-001// judges-ignore-next-line CYBER-002const y = dangerousOperation();// judges-file-ignore DOC-*    ← suppress globally for this file

Supported comment styles: //, #, /* */. Supports comma-separated rule IDs and wildcards (*, SEC-*).

Auto-Fix Patches

Certain findings include machine-applicable patches in the patch field:

Patches include oldText, newText, startLine, and endLine for automated application.

Cross-Evaluator Deduplication

When multiple judges flag the same issue (e.g., both Data Security and Cybersecurity detect SQL injection on line 15), findings are automatically deduplicated. The highest-severity finding wins, and the description is annotated with cross-references (e.g., "Also identified by: CYBER-003").

Human Focus Guide

Every tribunal evaluation includes a humanFocusGuide that categorizes findings into three buckets for human reviewers:

Blind spots are detected from code characteristics: complex branching logic, external service calls, financial calculations, PII handling, state machines, and complex regex. The guide appears in CLI text/markdown output, JSON/SARIF output, and GitHub Action step summaries.

Regulatory Scope

Configure which regulatory frameworks apply to your project in .judgesrc:

{ "regulatoryScope": ["GDPR", "PCI-DSS", "SOC2"] }

Findings that cite ONLY out-of-scope frameworks are suppressed. Findings with no regulatory reference (general code quality) are always kept. Run judges list --frameworks to see all 17 supported frameworks (GDPR, CCPA, HIPAA, PCI-DSS, SOC2, SOX, COPPA, FedRAMP, NIST, ISO27001, ePrivacy, DORA, NIS2, EU-AI-Act, and more).

Self-Teaching Amendments

The LLM benchmark system auto-generates precision amendments for judges with high false-positive rates. Amendments are data-driven corrections injected into prompts that improve accuracy over successive benchmark runs.

The self-teaching loop:

1. Run benchmark → analyzer identifies judges below 70% precision
2. Generates targeted amendments (e.g., "Judge ERR: do not flag clean Express code with framework error middleware")
3. Next benchmark run loads amendments → precision improves
4. Run judges codify-amendments to bake amendments permanently into the distributed package

Taint Flow Analysis

The engine performs inter-procedural taint tracking to trace data from user-controlled sources (e.g., req.body, process.env) through transformations to security-sensitive sinks (e.g., eval(), exec(), SQL queries). Taint flows are used to boost confidence on true-positive findings and suppress false positives where sanitization is detected.

Positive Signal Detection

Code that demonstrates good practices receives score bonuses (capped at +15):

Framework-Aware Rules

Judges include framework-specific detection for Express, Django, Flask, FastAPI, Spring, ASP.NET, Rails, and more. Framework middleware (e.g., helmet(), express-rate-limit, passport.authenticate()) is recognized as mitigation, reducing false positives.

Cross-File Import Resolution

In project-level analysis, imports are resolved across files. If one file imports a security middleware module from another file in the project, findings about missing security controls are automatically adjusted with reduced confidence.

Scoring

Each judge scores the code from 0 to 100:

Verdict logic:

• FAIL — Any critical finding, or score < 60
• WARNING — Any high finding, any medium finding, or score < 80
• PASS — Score ≥ 80 with no critical, high, or medium findings

The overall tribunal score is the average of all 45 judges. The overall verdict fails if any judge fails.

Project Structure

judges/├── src/│   ├── index.ts              # MCP server entry point — tools, prompts, transport│   ├── api.ts                # Programmatic API entry point│   ├── cli.ts                # CLI argument parser and command router│   ├── types.ts              # TypeScript interfaces (Finding, JudgeEvaluation, etc.)│   ├── config.ts             # .judgesrc configuration parser and validation│   ├── errors.ts             # Custom error types (ConfigError, EvaluationError, ParseError)│   ├── language-patterns.ts  # Multi-language regex pattern constants and helpers│   ├── judge-registry.ts     # Unified JudgeRegistry — single source of truth for all judges│   ├── plugins.ts            # Plugin API façade (delegates to JudgeRegistry)│   ├── scoring.ts            # Confidence scoring and calibration│   ├── dedup.ts              # Finding deduplication engine│   ├── fingerprint.ts        # Finding fingerprint generation│   ├── comparison.ts         # Tool comparison benchmark data│   ├── cache.ts              # Evaluation result caching│   ├── calibration.ts        # Confidence calibration from feedback data│   ├── fix-history.ts        # Auto-fix application history tracking│   ├── ast/                  # AST analysis engine (built-in, no external deps)│   │   ├── index.ts          # analyzeStructure() — routes to correct parser│   │   ├── types.ts          # FunctionInfo, CodeStructure interfaces│   │   ├── tree-sitter-ast.ts    # Tree-sitter WASM parser (all 8 languages)│   │   ├── structural-parser.ts  # Fallback scope-tracking parser│   │   ├── cross-file-taint.ts   # Cross-file taint propagation analysis│   │   └── taint-tracker.ts      # Single-file taint flow tracking│   ├── evaluators/           # Analysis engine for each judge│   │   ├── index.ts          # evaluateWithJudge(), evaluateWithTribunal(), evaluateProject(), etc.│   │   ├── shared.ts         # Scoring, verdict logic, markdown formatters│   │   └── *.ts              # One analyzer per judge (45 files)│   ├── formatters/           # Output formatters│   │   ├── sarif.ts              # SARIF 2.1.0 output│   │   ├── html.ts               # Self-contained HTML report (dark/light theme, filters)│   │   ├── junit.ts              # JUnit XML output (Jenkins, Azure DevOps, GitHub Actions)│   │   ├── codeclimate.ts        # CodeClimate/GitLab Code Quality JSON│   │   ├── diagnostics.ts        # Diagnostics formatter│   │   └── badge.ts              # SVG and text badge generator│   ├── commands/             # CLI subcommands│   │   ├── init.ts               # Interactive project setup wizard│   │   ├── fix.ts                # Auto-fix patch preview and application│   │   ├── watch.ts              # Watch mode — re-evaluate on save│   │   ├── report.ts             # Project-level local report│   │   ├── hook.ts               # Pre-commit hook install/uninstall│   │   ├── ci-templates.ts       # GitLab, Azure, Bitbucket CI templates│   │   ├── diff.ts               # Evaluate unified diff (git diff)│   │   ├── deps.ts               # Dependency supply-chain analysis│   │   ├── baseline.ts           # Create baseline for finding suppression│   │   ├── completions.ts        # Shell completions (bash/zsh/fish/PowerShell)│   │   ├── docs.ts               # Per-judge rule documentation generator│   │   ├── feedback.ts           # False-positive tracking & finding feedback│   │   ├── benchmark.ts          # Detection accuracy benchmark suite│   │   ├── rule.ts               # Custom rule authoring wizard│   │   ├── language-packs.ts     # Language-specific rule pack presets│   │   └── config-share.ts       # Shareable team/org configuration│   ├── presets.ts            # Named evaluation presets (strict, lenient, security-only, …)│   ├── patches/│   │   └── index.ts              # 201 deterministic auto-fix patch rules│   ├── tools/                # MCP tool registrations│   │   ├── register.ts           # Tool registration orchestrator│   │   ├── register-evaluation.ts    # Evaluation tools (evaluate_code, etc.)│   │   ├── register-workflow.ts      # Workflow tools (app builder, reports, etc.)│   │   ├── prompts.ts            # MCP prompt registrations (per-judge prompts)│   │   └── schemas.ts            # Zod schemas for tool parameters│   ├── reports/│   │   └── public-repo-report.ts   # Public repo clone + full tribunal report generation│   └── judges/               # Judge definitions (id, name, domain, system prompt)│       ├── index.ts          # Side-effect imports + re-exports (JUDGES, getJudge, getJudgeSummaries)│       └── *.ts              # One self-registering definition per judge (45 files)├── scripts/│   ├── generate-public-repo-report.ts  # Run: npm run report:public-repo -- --repoUrl <url>│   ├── daily-popular-repo-autofix.ts   # Run: npm run automation:daily-popular│   └── debug-fp.ts                     # Debug false-positive findings├── examples/│   ├── sample-vulnerable-api.ts  # Intentionally flawed code (triggers all judges)│   ├── demo.ts                   # Run: npm run demo│   └── quickstart.ts             # Quick-start evaluation example├── tests/│   ├── judges.test.ts            # Core judge evaluation tests│   ├── negative.test.ts          # Negative / FP-avoidance tests│   ├── subsystems.test.ts        # Subsystem integration tests│   ├── extension-logic.test.ts   # VS Code extension logic tests│   └── tool-routing.test.ts      # MCP tool routing tests├── grammars/                 # Tree-sitter WASM grammar files│   ├── tree-sitter-typescript.wasm│   ├── tree-sitter-cpp.wasm│   ├── tree-sitter-python.wasm│   ├── tree-sitter-go.wasm│   ├── tree-sitter-rust.wasm│   ├── tree-sitter-java.wasm│   └── tree-sitter-c_sharp.wasm├── judgesrc.schema.json      # JSON Schema for .judgesrc config files├── server.json               # MCP Registry manifest├── package.json├── tsconfig.json└── README.md

Scripts

Daily Popular Repo Automation

This repo includes a scheduled workflow at .github/workflows/daily-popular-repo-autofix.yml that:

• selects up to 10 repositories per day from a default pool of 100+ popular repos (or a manually supplied target),
• runs the full Judges evaluation across supported source languages,
• applies only conservative, single-line remediations that reduce matching finding counts,
• opens up to 5 PRs per repository with attribution to both Judges and the target repository,
• skips repositories unless they are public and PR creation is possible with existing GitHub auth (no additional auth flow).
• enforces hard runtime caps of 10 repositories/day and 5 PRs/repository.

Each run writes daily-autofix-summary.json (or SUMMARY_PATH) with per-repository telemetry, including:

• runAggregate — compact run-level totals and cross-repo top prioritized rules,
• runAggregate.totalCandidatesDiscovered and runAggregate.totalCandidatesAfterLocationDedupe — signal how much overlap was removed before attempting fixes,
• runAggregate.totalCandidatesAfterPriorityThreshold — candidates that remain after applying minimum priority score,
• runAggregate.dedupeReductionPercent — percent reduction from location dedupe for quick runtime-efficiency tracking,
• runAggregate.priorityThresholdReductionPercent — percent reduction from minimum-priority filtering after dedupe,
• priorityRulePrefixesUsed — dangerous rule prefixes used during prioritization,
• minPriorityScoreUsed — minimum candidatePriorityScore applied for candidate inclusion,
• candidatesDiscovered, candidatesAfterLocationDedupe, and candidatesAfterPriorityThreshold — per-repo candidate counts after each filter stage,
• topPrioritizedRuleCounts — most common rule IDs among ranked candidates,
• topPrioritizedCandidates — top ranked candidate samples (rule, severity, confidence, file, line, priority score).

Optional runtime control:

• AUTOFIX_MIN_PRIORITY_SCORE — minimum candidate priority score required after dedupe (default: 0, disabled).

Required secret:

• JUDGES_AUTOFIX_GH_TOKEN — GitHub token with permission to fork/push/create PRs for target repositories.

Manual run:

gh workflow run "Judges Daily Full-Run Autofix PRs" -f targetRepoUrl=https://github.com/owner/repo

Programmatic API

Judges can be consumed as a library (not just via MCP). Import from @kevinrabun/judges/api:

import {  evaluateCode,  evaluateProject,  evaluateCodeSingleJudge,  getJudge,  JUDGES,  findingsToSarif,} from "@kevinrabun/judges/api";
// Full tribunal evaluationconst verdict = evaluateCode("const x = eval(input);", "typescript");console.log(verdict.overallScore, verdict.overallVerdict);
// Single judgeconst result = evaluateCodeSingleJudge("cybersecurity", code, "typescript");
// SARIF output for CI integrationconst sarif = findingsToSarif(verdict.evaluations.flatMap(e => e.findings));

Package Exports

SARIF Output

Convert findings to SARIF 2.1.0 for GitHub Code Scanning, Azure DevOps, and other CI/CD tools:

import { findingsToSarif, evaluationToSarif, verdictToSarif } from "@kevinrabun/judges/sarif";
const sarif = verdictToSarif(verdict, "src/app.ts");fs.writeFileSync("results.sarif", JSON.stringify(sarif, null, 2));

Custom Error Types

All thrown errors extend JudgesError with a machine-readable code property:

import { ConfigError, EvaluationError } from "@kevinrabun/judges/api";try {  evaluateCode(code, "typescript");} catch (e) {  if (e instanceof ConfigError) console.error("Config issue:", e.code);}

License

MIT