AgentTrust

io.github.assister-xyzPublished by GitHub account assister-xyzv0.1.2Updated Oct 24, 2026

Quality verification for AI agents and MCP servers. 6-axis scoring, adversarial probes.

VerifiedSTDIODesktop onlySecurity & MonitoringAI & ML

Overview

AI-generated overview

AgentTrust evaluates other AI agents and MCP servers for quality, scoring them across six axes and issuing signed attestations as proof.

What you can do
Run full quality evaluations of a target MCP server: manifest, functional tool calls and judge scoring across six weighted axes, including adversarial probes for prompt injection, PII leakage and hallucination. Fetch a fast cached or manifest-only score, or look up an existing score with freshness decay. Verify an AQVC attestation token and decode its payload. Behind the tools sit a battle arena with head-to-head ratings, adaptive IRT testing, and an HTTP API for evaluations, leaderboards, badges and attestations.
When to use it
Useful when you want a pre-trust check on a third-party agent or MCP server before giving it real tasks, or when you need a signed, verifiable quality credential for a server you publish.
Requirements
Runs as a local Python process (PyPI package mcp-agenttrust, stdio) or via Docker. Needs at least one LLM provider key (CEREBRAS_API_KEY or GROQ_API_KEY; the README lists further fallbacks) and a MongoDB connection URL (MONGODB_URL); the full stack also uses Redis. Network access to the target server and LLM providers is required.
Before you install
Requires secret credentials: CEREBRAS_API_KEY, GROQ_API_KEY and MONGODB_URL, plus optional Ed25519 attestation keys and a Solana wallet for x402 payments. Evaluations send data to third-party LLM providers and connect to the target server, which may expose it to adversarial probes. The README mentions USDC/SOL payment verification, so payment-related configuration should be reviewed before use.

Installation

In SourceWeft

  1. Open AgentTrust in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

AgentTrust

Challenge-response quality verification for AI agents and MCP servers.

AgentTrust evaluates AI agent competency before you trust them with real tasks or payments. It connects to any MCP server, runs challenge-response tests across 6 quality dimensions, and issues W3C Verifiable Credentials as proof.

Why

The AI agent ecosystem has identity (ERC-8004, SATI), post-hoc reputation (TARS, Amiko), and payments (x402) — but no pre-payment quality gate. AgentTrust fills this gap: verify competency first, then trust.

Features

Evaluation Engine

  • 3-level pipeline: Manifest (schema) → Functional (tool calls) → Domain Expert (calibrated questions)
  • 6-axis scoring: accuracy (35%), safety (20%), reliability (15%), process quality (10%), latency (10%), schema quality (10%)
  • Consensus judging: 2-3 LLM judges in parallel with agreement threshold (saves 50-66% LLM calls)
  • 7 LLM provider fallback chain: Cerebras → Groq → OpenRouter → Gemini → Mistral → DeepSeek → OpenAI
  • 5 adversarial probe types: prompt injection, PII leakage, hallucination, overflow, system prompt extraction

Battle Arena

  • Head-to-head blind evaluation with position-swap consistency
  • OpenSkill (Bayesian ELO) rating system with divisions (Bronze → Grandmaster)
  • Fair matchmaking: rating proximity + uncertainty bonus + cross-division challenges
  • Style control penalties to prevent gaming via verbose/formatted responses

IRT Adaptive Testing

  • Rasch 1PL calibration from battle data (pure Python, no numpy)
  • Fisher information maximization for adaptive question selection
  • EAP ability estimation with standard normal prior
  • Reduces evaluation cost by 50-90% while maintaining accuracy

Standards

  • W3C Verifiable Credentials (AQVC format) with Ed25519 DataIntegrityProof
  • Google A2A v0.3 native support (AgentTrust IS an A2A agent)
  • x402 Solana payment verification (USDC + SOL)
  • AIUC-1 protocol mapping

Quick Start

Docker (recommended)

bash
cp .env.example .env# Add at least one LLM key (GROQ_API_KEY, CEREBRAS_API_KEY, etc.)docker compose up -d

Services:

Local Development

bash
python -m venv .venv && source .venv/bin/activatepip install -r requirements.txtcp .env.example .env# Add LLM keys to .env
unset GROQ_API_KEY  # Shell env overrides .env rotation poolpython -m uvicorn src.main:app --host 0.0.0.0 --port 8002 --reload

MCP Server (for Claude, Cursor, Windsurf)

Add to your MCP client config:

json
{  "mcpServers": {    "agenttrust": {      "command": "python",      "args": ["-m", "src.standards.mcp_server"],      "env": {        "GROQ_API_KEY": "your-key"      }    }  }}

Or connect to a running instance via SSE:

http://localhost:8003/sse

Available MCP tools:

ToolDescription
check_quality(server_url)Full evaluation: manifest + functional + judge scoring
check_quality_fast(server_url)Cached score (<10ms) or manifest-only (<100ms)
get_score(server_url)Lookup cached score with freshness decay
verify_attestation(attestation_jwt)Verify AQVC JWT and decode payload

API Endpoints

MethodEndpointDescription
POST/v1/evaluateSubmit target for evaluation
GET/v1/evaluate/{id}Poll evaluation status
GET/v1/score/{target_id}Get quality score
GET/v1/scoresSearch/list scores
GET/v1/badge/{target_id}.svgSVG quality badge
GET/v1/attestation/{id}Get signed attestation (JWT or W3C VC)
POST/v1/attestation/{id}/verifyVerify attestation
POST/v1/feedbackSubmit production feedback (anti-sandbagging)
POST/v1/battlesCreate evaluation battle
GET/v1/arena/leaderboardBattle arena leaderboard
GET/v1/rankingsGlobal rankings by domain/tier
POST/v1/irt/calibrateTrigger IRT batch calibration
GET/v1/irt/recommendAdaptive question selection
GET/v1/pricingx402 pricing table
GET/.well-known/agent.jsonA2A Agent Card

Architecture

src/  api/v1/          # 14 FastAPI routers  core/            # Evaluator, MCP client, scoring, IRT, battle arena  auth/            # API keys (SHA256 + salt), rate limiting by tier  storage/         # MongoDB (Motor) + Redis  payments/        # x402 protocol, Solana verification  standards/       # W3C VC issuer, A2A extension, MCP server, AIUC-1

Stack: FastAPI + MongoDB + Redis | 533 tests | 60 source files | 15 lean dependencies

Tests

bash
python -m pytest tests/ -q# 533 passed in ~2s

Configuration

See .env.example for all 60+ configuration options including:

  • LLM API keys (7 providers, comma-separated for rotation)
  • MongoDB/Redis connection
  • JWT attestation (Ed25519 key, issuer DID, validity)
  • Solana wallet for x402 payments
  • Rate limit tiers and consensus judge settings

License

MIT

Links

Built by Assisterr

Source: README.md at commit 6152a08

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.1.2LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts