Conkurrence

io.github.AlligatorC0derPublished by GitHub account AlligatorC0derv1.0.5Updated Oct 24, 2026

Multi-model rating for AI evaluation. Agreement is reliability, not validity. Maintenance mode.

VerifiedSTDIODesktop onlyData & AnalyticsAI & ML

Overview

AI-generated overview

Local MCP server that runs multi-model AI evaluations and reports inter-rater agreement statistics to flag contested items for expert review.

What you can do
Run an evaluation schema across multiple AI raters and compute agreement statistics such as Fleiss' kappa and Kendall's W against expert anchor items, with a self-consistency mode that uses the host model instead of API keys. Generate detailed markdown reports of a run. Compare two runs side by side and track agreement trends across runs to detect degradation. Suggest an evaluation schema from your data and validate a schema before running. Estimate cost and token usage before an evaluation.
When to use it
Useful when you are building or auditing an AI evaluation and want to know where model raters agree or disagree, which items need expert judgment, and whether disagreement stems from the criteria or the raters. It is a diagnostic aid, not a substitute for expert-labelled ground truth.
Requirements
Runs as a local process via npx or a global npm install, so Node.js is needed. Multi-model evaluation against Bedrock, OpenAI and Gemini models requires the corresponding provider credentials; the self-consistency mode needs no API keys and uses the host model through MCP Sampling. Desktop MCP client such as Claude Desktop.
Before you install
Multi-model runs call external providers and incur costs, so use the cost estimation tool first. The project is in maintenance mode, not accepting purchases and with no support commitment. Agreement measures reliability, not validity: models can agree confidently on a wrong answer, so results should not replace expert-labelled ground truth.

Installation

In SourceWeft

  1. Open Conkurrence in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

ConKurrence

Multi-model rating for AI evaluation: measures agreement among model raters (Fleiss' κ, Kendall's W) to find contested items for expert review.

What agreement tells you: where your raters and your criteria are consistent, and where they are not — which items are contested, and whether disagreement comes from the criteria or the raters.

What it does not tell you: that the raters are right. Agreement is reliability, not validity. Models can agree confidently on a wrong answer. Use ConKurrence to decide where an expert's judgment is needed and to diagnose your evaluation criteria — not to replace expert-labelled ground truth.

Status: maintenance mode. ConKurrence is developed by JE Vectors LLC as the evaluation instrument behind Inqura. It is not accepting purchases and carries no support commitment.

Install

bash
npm install -g conkurrence

MCP Server

Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:

bash
npx conkurrence mcp

Claude Desktop Configuration

Add to your claude_desktop_config.json:

json
{  "mcpServers": {    "conkurrence": {      "command": "npx",      "args": ["-y", "conkurrence", "mcp"]    }  }}

Claude Code Plugin

/plugin marketplace add AlligatorC0der/conkurrence

Features

  • Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously
  • Agreement statistics — Fleiss' kappa with bootstrap confidence intervals; Kendall's W against expert-labelled anchor items
  • Self-consistency mode — No API keys needed; uses the host model via MCP Sampling
  • Schema suggestion — AI-powered schema design from your data
  • Trend tracking — Compare runs over time, detect agreement degradation
  • Cost estimation — Know the cost before running

MCP Tools

ToolDescription
conkurrence_runExecute an evaluation across multiple AI raters
conkurrence_reportGenerate a detailed markdown report
conkurrence_compareSide-by-side comparison of two runs
conkurrence_trendTrack agreement over multiple runs
conkurrence_suggestAI-powered schema suggestion from your data
conkurrence_validate_schemaValidate a schema before running
conkurrence_estimateEstimate cost and token usage

Links

License

BUSL-1.1 — Business Source License 1.1

Source: README.md at commit 72e7732

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

2
  1. v1.0.5LatestOct 3, 2026
  2. v1.0.3Sep 16, 2026

Score

40% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts

    Declare a license in the repository or the registry entry.