ML Referee

io.github.scarricoPublished by GitHub account scarricov1.1.0Updated Oct 29, 2026

Evaluate ML features beyond a baseline and test bounded feature repairs. Invite-only beta.

VerifiedStreamable HTTPWeb executableData & AnalyticsAI & ML

Overview

AI-generated overview

Remote MCP server for ML Referee that lets an assistant upload de-identified datasets, quote and run feature audits against a baseline, and test bounded…

What you can do
  • Discover the service, its privacy terms, supported evaluation profiles and readiness state
  • Upload encrypted, de-identified datasets, campaign shards or repair matrices and preflight them without spending credits
  • Quote audits, repairs and campaigns, then submit an accepted quote and track audit status
  • Read completed tenant-private results, signed output manifests and report receipts, and verify a report hash
  • Record audit follow-ups and delete retained uploads or audits
  • Check audit-credit balance, redeem a beta invite or request beta access
When to use it
Useful when you want an assistant to check whether ML features add value beyond a baseline, or to diagnose degradation and test bounded feature repairs on private data, with quoting and credit tracking handled through the same connection. It is an invite-only beta, so it fits teams already granted or seeking access.
Requirements
A remote streamable HTTP endpoint at api.mlreferee.com; no local package or environment variables are declared. The tools refer to an authenticated tenant, credentials and audit credits, so an account and beta access are needed. Datasets are uploaded as de-identified CSV or JSON payloads.
Before you install
Uploads send data to a third party, even though they are described as encrypted and de-identified; confirm what may leave your environment. Audits consume paid audit credits, and submitting an audit reserves them. Tools can delete retained uploads and audit results. Beta invites are operator-issued and expire on January 1, 2027.

Installation

In SourceWeft

  1. Open ML Referee in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.scarrico%2Fmlreferee/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "ml-referee": {
      "type": "http",
      "url": "https://api.mlreferee.com/mcp"
    }
  }
}

Tools

26

describe_referee

Describe Referee

Discover capabilities, privacy, and the approved public catalog.

Takes no parameters.

list_verticals

List evaluation profiles

List the three supported evaluation profiles and their discovery tags.

Takes no parameters.

get_referee_readiness

Check Referee readiness

Report connection, credential, audit-credit, and submission readiness with the next action.

Takes no parameters.

get_referee_usage

Read aggregate usage

Read aggregate, privacy-preserving Referee activity and outcome evidence for the selected audience.

Parameterswindow_days

get_input_schema

Get feature-audit schema

Get neutral column roles and validation defaults for a vertical.

Parametersverticalrequired

get_repair_schema

Get feature-repair schema

Get the feature-only contract for diagnosing degradation and testing bounded repairs.

Takes no parameters.

upload_private_dataset

Upload private dataset

Encrypt a de-identified CSV/JSON payload for a short tenant-private retention window.

Parametersdatarequiredformatretention_seconds

upload_campaign_shard

Upload campaign shard

Encrypt one of up to 64 private feature shards for a large campaign; retain for at least 4,200 seconds.

Parametersdatarequiredformatretention_seconds

upload_repair_matrix

Upload repair matrix

Encrypt one complete de-identified repair matrix of up to 100 MB and 1,000 diagnostic features.

Parametersdatarequiredformatretention_seconds

preflight_dataset

Preflight feature audit

Validate a tenant-private upload and its baseline evidence/row identity without spending a credit.

Parameterscontractrequiredverticalrequiredupload_idrequired

preflight_repair

Preflight feature repair

Validate a tenant-private feature-repair matrix without spending a credit.

Parameterscontractrequiredupload_idrequired

quote_audit

Quote feature audit

Create a contract-bound quote for one audit credit; baseline mode and evidence cannot be substituted.

Parameterscontractrequiredverticalrequiredupload_idrequiredidempotency_keyrequired

quote_repair

Quote feature repair

Preflight and quote label-based feature repair; large jobs return a measured ETA and credit requirement.

Parameterscontractrequiredupload_idrequiredidempotency_keyrequired

quote_campaign

Quote feature campaign

Preflight all private feature shards, time a pilot, and quote credits and ETA for up to 1,000 candidates in one corrected search family.

Parametersshardsrequiredverticalrequiredidempotency_keyrequired

get_credits

Read audit credits

Read beta or paid audit-credit availability and usage.

Takes no parameters.

redeem_beta_invite

Redeem beta invite

Redeem one operator-issued invite for free beta audit credits before January 1, 2027.

Parametersinvite_coderequired

request_beta_access

Request beta access

Request operator-reviewed beta access without purchasing or granting credits.

Parameterslanerequireduse_caserequiredidempotency_keyrequired

get_beta_request_status

Get beta request status

Read the authenticated tenant's beta-access request status.

Parametersrequest_idrequired

submit_audit

Submit audit

Accept a quote and atomically reserve its disclosed audit credits.

Parametersquote_idrequiredidempotency_keyrequired

record_audit_followup

Record audit follow-up

Record whether a completed audit changed a decision or was confirmed prospectively.

Parameterskindrequiredvaluerequiredaudit_idrequired

get_audit_status

Get audit status

Read the state of an audit owned by the authenticated tenant.

Parametersaudit_idrequired

get_audit_result

Get audit result

Read a completed tenant-private result and signed output manifest.

Parametersaudit_idrequired

get_audit_receipt

Get report receipt

Read a tenant-private report fingerprint retained after report expiry or deletion. Older reports may have no record. This does not retrieve report contents.

Parametersaudit_idrequired

verify_audit_report

Verify report fingerprint

Compare a locally computed SHA-256 with the retained tenant-private receipt. Use the exact downloaded result JSON bytes (not the API envelope), or UTF-8 result.report text with representation report_text. Never copy the expected hash as the candidate: compute it from the supplied artifact. A digest match does not validate a summary, factual correctness, or absence of harm. No report is uploaded.

Parameterssha256requiredaudit_idrequiredrepresentation

delete_audit

Delete audit

Delete retained inputs and results for an audit owned by this tenant.

Parametersaudit_idrequired

delete_upload

Delete upload

Delete a tenant-private upload that is no longer needed.

Parametersupload_idrequired

Version history

1
  1. v1.1.0LatestOct 29, 2026

Score

83% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts

    Declare a GitHub repository with a README, or serve resource_documentation in the OAuth protected-resource metadata.

  • LicenseNot applicable

    A hosted service is used under its provider's terms, so no license is expected here. This check is left out of the percentage.