Backtest Overfitting Audit

io.github.tylerscomic-labv1.0.0Updated Oct 2, 2026

Probability of Backtest Overfitting (CSCV), Deflated Sharpe Ratio, and purged CV splits.

VerifiedStreamable HTTPWeb executableFinanceData & Analytics

Overview

AI-generated overview

Audits trading backtests for overfitting using CSCV probability, Deflated Sharpe Ratio, minimum backtest length, and purged CV splits.

What it does
This server implements statistical checks on strategy backtests. Its tools estimate the Probability of Backtest Overfitting via Combinatorially Symmetric Cross-Validation, compute a Deflated Sharpe Ratio adjusted for the number of trials and non-normal returns, report the minimum backtest length needed for a Sharpe ratio to be meaningful, and generate purged and embargoed cross-validation splits for time-series data. It targets the failure mode where testing many parameter variants produces a good-looking result by chance.
When to use it
Use it when evaluating a quantitative trading strategy and you need to know whether an apparent edge survives out-of-sample, or when you have run many parameter combinations against the same history. It is also useful for building leakage-free cross-validation splits for time-series backtests.
Requirements
A remote streamable HTTP endpoint; no authentication, environment variables, or headers are declared. The README also describes a self-hosted option via npm install and node server.js, which requires Node.js. The hosted option is described as having a free tier and a paid Pro tier.
Before you install
The README mentions a free tier and a paid Pro tier for higher limits, so hosted use may involve payment or usage limits. Backtest data submitted for analysis is sent to the remote endpoint. No credentials are requested by the manifest.

Installation

In SourceWeft

  1. Open Backtest Overfitting Audit in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "overfitting-audit-mcp": {
      "type": "http",
      "url": "https://overfitting-audit-mcp.mcpize.run/mcp"
    }
  }
}

README

overfitting-audit-mcp

[License: MIT] [Live on MCPize]

An MCP server that answers "is this edge real, or a testing-hundreds-of-variants artifact?" — implementing the Probability of Backtest Overfitting (CSCV method), Deflated Sharpe Ratio, Minimum Backtest Length, and purged/embargoed cross-validation splits.

The problem this solves

Testing enough parameter combinations against the same historical data will eventually produce a great-looking backtest by chance alone. Standard backtest metrics (Sharpe, win rate, profit factor) don't distinguish a genuine edge from the best-looking result out of hundreds of near-identical variants. This audits for that specific failure mode directly, rather than trusting a single strong-looking curve.

Tools

probability_of_backtest_overfitting

Combinatorially Symmetric Cross-Validation (CSCV) method — estimates the probability that a strategy's in-sample performance rank won't hold out-of-sample.

deflated_sharpe_ratio

Adjusts a Sharpe ratio for the number of trials run and the non-normality of returns, so it can't be inflated just by testing more variants.

minimum_backtest_length

The minimum number of independent trials/observations needed before a given Sharpe ratio is statistically meaningful at all.

purged_cv_split

Generates purged and embargoed cross-validation splits for time-series backtests, preventing the lookahead leakage that ordinary k-fold CV introduces on financial data.

Use it

Hosted (recommended): MCPize — free tier, paid Pro tier for higher limits.

Self-host:

bash
npm installnode server.js

Part of the AlgoForge suite

Prop-firm and quant-validation tools for algo traders: prop-rules-mcp, trade-journal-mcp, payout-calc-mcp, econ-calendar-mcp, montecarlo-validator-mcp, walkforward-validator-mcp, pinescript-audit-mcp, backtest-cost-sensitivity-mcp, pinescript-mcp.

License

MIT

Source: README.md at commit c0087b1

Tools

0
Tool metadata has not been indexed yet.

Version history

1
  1. v1.0.0LatestOct 2, 2026