A/B Test Sample Size Calculator MCP

io.github.theluckystrikePublished by GitHub account theluckystrikev1.0.0Updated Nov 3, 2026

Exact A/B test sample size, duration, z-test, chi-square, Fisher-Freeman-Halton and peeking risk.

The publisher's own server: Calculator Tools MCP

VerifiedStreamable HTTPWeb executable

Overview

A/B Test Sample Size Calculator's hosted MCP server (ab-test-sample-size-calculator-mcp.lipmichal.workers.dev); no sign-in required; Streamable HTTP; 5 tools.

No AI overview yet: these facts come from the registry entry and the server itself.

Endpoint
ab-test-sample-size-calculator-mcp.lipmichal.workers.dev
Last checked

Installation

In SourceWeft

  1. Open A/B Test Sample Size Calculator MCP in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Web executable via Streamable HTTP. Remote servers run from the web runtime once configured in a workspace.

Use with an agent

Paste this prompt into Claude Code, Codex, Cursor or another agent — or into SourceWeft chat. The agent reads this server's install guide, shows you its publisher, endpoint and sign-in, and adds it to your client once you agree. Keys never go into a command.

Read https://sourceweft.com/mcp/io.github.theluckystrike%2Fab-test-sample-size-calculator-mcp/install.md and connect the MCP server it describes to my client. Before changing any config, show me its publisher, endpoint and sign-in method and wait for my OK. Never put an API key in a command or in a file I would commit; ask me where it should come from.

Read the install guide the agent follows

Other MCP clients

Add this to your client's mcpServers config.

{
  "mcpServers": {
    "a-b-test-sample-size-calculator": {
      "type": "http",
      "url": "https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp"
    }
  }
}

README

ab-test-sample-size-calculator-mcp

A/B test sample size calculator and significance tests for AI assistants, with exact answers instead of mental arithmetic.

This is a Model Context Protocol (MCP) server. It works out how many users an A/B test needs, how long it must run, whether a result is significant (z-test, chi-square, Fisher exact), the Fisher-Freeman-Halton exact test for larger tables, and how much peeking inflates false positives. It needs no API key and makes no network calls.

Tools

ToolWhat it returns
calculate_sample_sizeUsers per variant for a conversion-rate test: baseline, relative or absolute MDE, alpha, power, one or two-sided, several variants with optional Bonferroni, unequal splits. Also the figure Evan Miller's calculator gives for the same inputs.
estimate_test_durationDays and whole weeks to reach the sample size, from daily visitors, traffic share and number of variants.
test_ab_significanceTwo-proportion z-test, p-value, confidence interval, relative lift, chi-square with and without Yates, Fisher exact p and a sample ratio mismatch check.
fisher_exact_testExact p-value for a 2x2 table (two-sided, greater, less) or any r x c table (Fisher-Freeman-Halton), next to Pearson chi-square and an expected-count check.
check_peeking_riskThe real false positive rate when a test is checked K times, and Pocock and O'Brien-Fleming boundaries for up to 10 looks.

Every result has structured JSON, a one-paragraph summary, as_of, source (the method references) and more (the matching calculator on abwex.com).

What it knows that your model does not

  • Exact inverse normal quantiles and the full sample size formula. Models round z-values, drop the pooled variance term or confuse relative and absolute MDE, and the answer moves by thousands of users.
  • Why calculators disagree. The result shows both the standard pooled formula (as R power.prop.test) and Evan Miller's published formula, so a 37,513 vs 36,314 gap has an explanation.
  • Exact hypergeometric sums. A Fisher-Freeman-Halton p-value for a 4 x 4 table needs millions of tables summed; it cannot be estimated in your head.
  • Sequential testing numbers: checking 5 times at p < 0.05 gives a 14.2% false positive rate, and the boundaries that fix it.

Install

Hosted (Claude.ai custom connector, ChatGPT developer mode)

Paste this URL. No token, no sign-up.

https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp

Claude Code

claude mcp add --transport http ab-test-calculator https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp

Or run it locally over stdio:

claude mcp add ab-test-calculator -- npx -y github:theluckystrike/ab-test-sample-size-calculator-mcp

Claude Desktop

Add to claude_desktop_config.json:

json
{  "mcpServers": {    "ab-test-calculator": { "command": "npx", "args": ["-y", "github:theluckystrike/ab-test-sample-size-calculator-mcp"] }  }}

Cursor

Add to ~/.cursor/mcp.json:

json
{  "mcpServers": {    "ab-test-calculator": { "url": "https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp" }  }}

VS Code

Add to .vscode/mcp.json:

json
{  "servers": {    "ab-test-calculator": { "type": "http", "url": "https://ab-test-sample-size-calculator-mcp.lipmichal.workers.dev/mcp" }  }}

Node 20 or newer is needed for the local stdio version.

Example prompts

  • "Our checkout converts at 4.2%. How many users per variant to detect a 10% relative lift with 80% power?"
  • "We get 4,000 visitors a day and want to test 3 variants. How many weeks will it take?"
  • "Control had 420 conversions from 10,000, the variant 495 from 10,050. Is that significant?"
  • "Run a Fisher-Freeman-Halton test on [[12, 30, 8], [20, 25, 15]]."
  • "If I check the dashboard every day for 10 days, what is my real false positive rate?"

Data sources

No live data. Methods and the published values the tests check against are in DATA_SOURCES.md: R power.prop.test and fisher.test documentation outputs, Evan Miller's calculator code, scipy and statsmodels reference values (scripts/reference.py), Armitage et al. (1969) and Jennison and Turnbull (2000).

Limitations

  • Sample size covers conversion rates (two proportions). Continuous metrics such as revenue per user need the standard deviation and are not covered.
  • Sample size and duration assume a fixed-horizon test analysed once. For several looks, plan with check_peeking_risk.
  • The Fisher-Freeman-Halton test enumerates tables exactly. Very large or sparse r x c tables can exceed the work limit (lower on the hosted server than locally). The tool then returns the chi-square result with a note instead of an exact p.
  • Peeking boundaries assume equally spaced looks and two-sided tests; boundaries are given for up to 10 looks.
  • The confidence interval for the difference is the Wald interval, which is poor for rates near 0 or 1 with small samples.

More

Interactive versions of these calculators and guides are on abwex.com, including the Fisher-Freeman-Halton calculator and the peeking inflation calculator.

Part of the Zovo MCP collection: https://mcp.zovo.one

Development

npm installnpm test                      # builds, then runs the fixture testspython3 scripts/reference.py  # recomputes the reference values with scipy and statsmodelsnode scripts/smoke-stdio.mjs  # stdio check, one call per toolnpx wrangler deploy           # Cloudflare Worker, Streamable HTTP at /mcp

License

MIT

More data servers from the same author

Each one is free, needs no API key and has a hosted URL you can paste into Claude or ChatGPT.

servertoolshosted URL
cups-to-grams-mcpconvert_ingredient, search_ingredients, scale_recipe, convert_oven_temperaturehttps://cups-to-grams-mcp.lipmichal.workers.dev/mcp
dscr-loan-calculator-mcpcalculate_dscr, max_loan_for_target_dscr, rent_needed_for_target_dscr, compare_scenarios, amortization_summaryhttps://dscr-loan-calculator-mcp.lipmichal.workers.dev/mcp
gold-price-per-gram-mcpget_gold_price_per_gram, get_metal_prices, calculate_jewelry_value, convert_gold_unitshttps://gold-price-per-gram-mcp.lipmichal.workers.dev/mcp
lake-level-mcpsearch_lakes, get_lake_level, get_lake_history, compare_lakeshttps://lake-level-mcp.lipmichal.workers.dev/mcp
minimum-wage-mcpget_minimum_wage, list_upcoming_increases, compare_states, annual_income_at_minimum_wagehttps://minimum-wage-mcp.lipmichal.workers.dev/mcp
no-tax-on-overtime-mcpcalculate_overtime_deduction, calculate_tips_deduction, calculate_senior_deduction, estimate_tax_savings, explain_eligibilityhttps://no-tax-on-overtime-mcp.lipmichal.workers.dev/mcp
paycheck-calculator-mcpcalculate_take_home_pay, compare_states_take_home, salary_to_hourly, hourly_to_salary, get_federal_tax_bracketshttps://paycheck-calculator-mcp.lipmichal.workers.dev/mcp
uk-income-tax-salary-mcpcalculate_uk_take_home, get_uk_tax_bands, compare_salaries, marginal_rate_at_incomehttps://uk-income-tax-salary-mcp.lipmichal.workers.dev/mcp
us-federal-holidays-business-days-mcplist_federal_holidays, is_business_day, add_business_days, count_business_days, invoice_due_datehttps://us-federal-holidays-business-days-mcp.lipmichal.workers.dev/mcp
aca-premium-tax-credit-calculator-mcpget_federal_poverty_level, calculate_premium_tax_credit, check_subsidy_cliff, get_aptc_repayment_caphttps://aca-premium-tax-credit-calculator-mcp.lipmichal.workers.dev/mcp
ap-score-distributions-mcpget_ap_score_distribution, estimate_ap_percentile, compare_ap_exams, list_ap_exams, estimate_ap_score_from_compositehttps://ap-score-distributions-mcp.lipmichal.workers.dev/mcp
car-loan-interest-deduction-calculator-mcpcalculate_car_loan_interest_deduction, calculate_salt_deduction, get_tax_brackets_and_standard_deduction, calculate_child_tax_credithttps://car-loan-interest-deduction-calculator-mcp.lipmichal.workers.dev/mcp
estate-gift-tax-exemption-calculator-mcpget_exemption_amounts, calculate_estate_tax, check_gift, compare_yearshttps://estate-gift-tax-exemption-calculator-mcp.lipmichal.workers.dev/mcp
recipe-nutrition-calculator-mcplookup_ingredient_nutrition, calculate_recipe_nutrition, round_nutrition_label, scale_recipe_nutritionhttps://recipe-nutrition-calculator-mcp.lipmichal.workers.dev/mcp
retirement-contribution-limits-2027-mcpget_contribution_limits, calculate_max_contribution, get_social_security_cola, compare_yearshttps://retirement-contribution-limits-2027-mcp.lipmichal.workers.dev/mcp
section-179-deduction-calculator-mcpcalculate_section_179, calculate_bonus_depreciation, get_vehicle_depreciation_limits, calculate_asset_loan_paymenthttps://section-179-deduction-calculator-mcp.lipmichal.workers.dev/mcp
student-loan-repayment-assistance-plan-calculator-mcpcalculate_rap_payment, compare_rap_vs_standard, get_federal_loan_limits, get_plan_transition_dateshttps://student-loan-repayment-assistance-plan-calculator-mcp.lipmichal.workers.dev/mcp
subdomain-finder-mcpfind_subdomains, resolve_subdomains, get_certificate_history, summarize_attack_surfacehttps://subdomain-finder-mcp.lipmichal.workers.dev/mcp
uk-statutory-sick-pay-maternity-pay-calculator-mcpcalculate_statutory_sick_pay, calculate_statutory_maternity_pay, get_minimum_wage_rates, check_eligibilityhttps://uk-statutory-sick-pay-maternity-pay-calculator-mcp.lipmichal.workers.dev/mcp

The paid back-office collection (invoices, VAT, PDFs, spreadsheets) is at https://mcp.zovo.one

Source: README.md at commit a7040c5

Tools

5

calculate_sample_size

A/B test sample size calculator

Exact sample size per variant for an A/B test on a conversion rate (two independent proportions): baseline rate, minimum detectable effect (relative or absolute), alpha, power, one or two-sided, several variants with optional Bonferroni, and unequal allocation. Uses the pooled/unpooled normal formula (same as R power.prop.test) and also reports the figure Evan Miller's calculator gives, so the two can be reconciled.

Parametersmderequiredalphapowersidedmde_typevariantsbonferronibaseline_raterequiredallocation_ratio

estimate_test_duration

A/B test duration estimator

How many days an A/B test must run: from a known sample size per variant, or from baseline rate and MDE (computed as in calculate_sample_size), plus daily visitors and the share of traffic in the test. Returns exact days and the recommended whole weeks (to cover weekday effects).

Parametersmdealphapowermde_typevariantsbaseline_ratetraffic_sharedaily_visitorsrequiredsample_size_per_variant

test_ab_significance

A/B test significance calculator

Is an A/B test result statistically significant? From visitors and conversions per group: two-proportion z-test (pooled), one or two-sided p-value, Wald confidence interval for the difference, relative lift, Pearson chi-square with and without Yates correction (R prop.test), Fisher exact p for the same 2x2 table, and a sample ratio mismatch (SRM) check against the planned split.

Parametersalphaalternativeexpected_splitcontrol_visitorsrequiredvariant_visitorsrequiredcontrol_conversionsrequiredvariant_conversionsrequired

fisher_exact_test

Fisher exact and Fisher-Freeman-Halton test

Exact p-value for a contingency table of counts. 2x2: Fisher exact test (two-sided, greater or less, as R fisher.test). Larger r x c tables (for example 3 variants x converted/not, or 4 x 4 survey tables): Fisher-Freeman-Halton exact test by a network algorithm. Also returns Pearson chi-square and how many cells have expected counts below 5, so you can see when the chi-square approximation is unsafe.

Parameterstablerequiredalternative

check_peeking_risk

A/B test peeking and sequential testing risk

What peeking does to an A/B test: the real false positive rate if you check a fixed-horizon test K times and stop at the first p < alpha (for example about 14% at 5 looks with alpha 0.05), and the Pocock and O'Brien-Fleming group sequential boundaries (z and nominal p per look) that keep the overall rate at alpha, for up to 10 equally spaced looks.

Parametersalphalooksrequired

Version history

1
  1. v1.0.0LatestNov 3, 2026

Score

100% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 daysRequired25 pts
  • Auth declaredRequired10 pts
  • Tools listedRequired15 pts
  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts

Other Calculator integrations

Calculator Tools MCP

tools.calculatorOfficial publisher

A remote MCP endpoint that lets an assistant publish a live calculator, tracker, tool, or game from Cursor, Claude, or ChatGPT.

Streamable HTTPWeb executable
Productivity & WorkflowNot indexed yetOct 25, 2026

Free IFTA Calculator

dev.pages.ifta-calculatorOfficial publisher

Read-only helper that answers US/Canadian IFTA fuel-tax questions and estimates quarterly tax from miles and gallons per jurisdiction.

Streamable HTTPWeb executable
Finance1 toolOct 29, 2026

Complexity Cost Calculator

Published by GitHub account beamersaas

Remote MCP server that prices software complexity and projects how AI adoption changes engineering cost over quarters.

Streamable HTTPWeb executableAuth
Data & Analytics2 toolsNov 6, 2026

UK Statutory Sick Pay and Maternity Pay

Published by GitHub account theluckystrike

UK Statutory Sick Pay and Maternity Pay's hosted MCP server; no sign-in required; Streamable HTTP; 4 tools.

Streamable HTTPWeb executable
4 toolsNov 3, 2026

Student Loan RAP Calculator

Published by GitHub account theluckystrike

Student Loan RAP Calculator's hosted MCP server; no sign-in required; Streamable HTTP; 4 tools.

Streamable HTTPWeb executable
4 toolsNov 3, 2026