AIForum.

How to Run Statistical Significance Tests on AI Evaluation Results

On this page

THE SHORT ANSWER

A score difference is not a result until you test it. For two models evaluated on the same items, use McNemar's exact test on the items where they disagree; for F1, BLEU, and other composite metrics, use a paired bootstrap or randomization test. Report the effect size, a confidence interval, and the p-value together — and be honest about small benchmarks.
Quick facts

If you changed a prompt, quantized a checkpoint, or swapped in a new model and the headline score moved by a fraction of a percent, you do not yet know whether anything improved. The practical answer is to treat the comparison as a paired statistical test: evaluate both systems on the same items, classify each item as an improvement, a regression, or a tie, and then test whether the number of changes is larger than sampling noise would produce. For accuracy-style metrics, that test is usually McNemar’s exact binomial test. For F1, BLEU, ROUGE, and other composite metrics, it is a paired bootstrap or approximate randomization test.

This guide is for practitioners who run evaluations on local models, hosted APIs, or both, and who need to defend a conclusion such as “the new checkpoint is better.” It assumes you already have per-item outputs from two systems on a shared test set. Everything below is about turning those outputs into a defensible claim rather than a guess.

Why a single average score is not a measurement

An evaluation score is a statistic computed from a sample of items. Swap the sample, or rerun a stochastic model, and the score moves. The NVIDIA NeMo Evaluator documentation makes the point bluntly: a small benchmark such as HumanEval, with 164 items, “can fluctuate by several percentage points between runs just from sampling noise,” and a single average score hides the fact that a candidate can trade improvements on some problems for regressions on others.

A significance test formalizes one specific question: if the two systems were equally good on every item, how often would we see a difference at least this large? The resulting p-value answers that question. It is not the probability that the new model is better, and it is not a measure of how much better it is. Those are separate quantities — the effect size and the confidence interval — and they belong in the same report.

Choosing a test: three questions

Dror et al.’s protocol in The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing reduces test selection to a few properties of your setup: how many systems you compare, whether they were evaluated on the same items, and whether the evaluation measure decomposes per item. The table below maps common situations onto a reasonable default.

Your setupReasonable defaultWhy
Two systems, same test set, per-item correctness (accuracy, exact match, pass/fail)McNemar’s exact binomial testUses the pairing; only the items where the systems disagree carry information
Two systems, same test set, composite metric (F1, BLEU, ROUGE)Paired bootstrap or approximate randomization testThe metric does not decompose into independent per-item successes, so a contingency table is not enough
Two systems, different test setsUnpaired test (e.g. Wilcoxon or a t-test on per-item or per-run scores)No pairing exists, so the test is weaker and the comparison is confounded with test-set differences
Many systems on one test setCorrected pairwise tests (Holm or Bonferroni) or an omnibus test firstRunning many uncorrected tests inflates the chance of a false positive
Multiple random seeds per systemPaired t-test or Wilcoxon signed-rank over runsTreats each seed as a paired observation of the same configuration

The NVIDIA documentation explains why a paired test is preferred over a plain t-test on the two aggregate scores: comparing two averages throws away the item-level pairing and treats a swap of one improvement for one regression as if nothing happened. A paired test analyzes exactly that swap.

Worked example: McNemar’s exact test for two models on the same items

McNemar’s test looks only at the discordant pairs — items where one system is correct and the other is wrong. Build a 2×2 table:

  • b: baseline correct, candidate wrong (a regression)
  • c: baseline wrong, candidate correct (an improvement)
  • concordant cells (both correct, both wrong) are ignored

Under the null hypothesis that the two systems are equally good, each discordant item is a fair coin flip, so b follows a binomial distribution with n = b + c and p = 0.5. The exact test is therefore a single binomial test.

from scipy.stats import binomtest

# b = baseline correct, candidate wrong  (regressions)
# c = baseline wrong,  candidate correct (improvements)
b, c = 37, 58
discordant = b + c

result = binomtest(b, discordant, 0.5, alternative="two-sided")
print(f"discordant pairs: {discordant}")
print(f"exact p-value: {result.pvalue:.4f}")

With b = 37 and c = 58 (95 discordant pairs), the exact two-sided p-value comes out to roughly 0.03. These are illustrative numbers, not a measured benchmark result — the point is the shape of the computation: you feed the test two counts, not two accuracy percentages, and you get back a probability that the observed imbalance arose by chance.

Two practical notes. First, the number of discordant pairs is your real sample size; with only a handful of disagreements, McNemar’s test has almost no power and a non-significant result means “unresolved,” not “equivalent.” Second, always report b and c alongside the p-value. A reviewer can then verify the arithmetic and see whether the improvement is broad or concentrated in a few items.

When the metric is not accuracy: the paired bootstrap

F1, macro-averaged scores, BLEU, and similar metrics cannot be reduced to a per-item coin flip, because changing one prediction can change the score in a way that depends on the whole corpus. The standard workaround is to resample the test items with replacement many times, recompute the metric difference on each resample, and use the resulting distribution to build a confidence interval and a p-value.

import numpy as np
from sklearn.metrics import f1_score

rng = np.random.default_rng(42)

def paired_bootstrap(y_true, pred_a, pred_b, n_resamples=10_000):
    n = len(y_true)
    deltas = np.empty(n_resamples)
    for i in range(n_resamples):
        idx = rng.integers(0, n, n)
        deltas[i] = (f1_score(y_true[idx], pred_a[idx], average="macro")
                     - f1_score(y_true[idx], pred_b[idx], average="macro"))
    lo, hi = np.percentile(deltas, [2.5, 97.5])
    # illustrative two-sided p-value: how often the sign flips
    p = 2 * min((deltas <= 0).mean(), (deltas >= 0).mean())
    return deltas.mean(), lo, hi, min(p, 1.0)

Two cautions. The p-value construction above is a common heuristic; for a formal treatment, follow the paired bootstrap test described in Dror et al., which specifies how to count resamples under the null. And seed the resampler (rng above) so that rerunning the analysis reproduces the same interval — an unseeded bootstrap is not reproducible.

An approximate randomization test is a close relative: instead of resampling items, you repeatedly swap the two systems’ outputs on random subsets of items and recompute the metric difference. Both approaches answer the same question and, on the same data, usually agree.

Reporting: effect size, interval, and p-value together

A p-value alone is a poor summary. The NVIDIA comparison tooling reports an effect size, a confidence interval, and a human-readable verdict alongside the test result, and that triad is a good minimum standard for any evaluation report:

  • Effect size — the actual difference in the metric (for example, +1.4 points of accuracy), stated in the units your readers care about.
  • Confidence interval — the range of differences compatible with the data. A wide interval crossing zero is a warning that the benchmark is too small to resolve the question.
  • p-value — the probability of seeing a difference this large if the systems were equally good.

If you compare many prompt variants or many checkpoints against one baseline, correct for multiple comparisons with a Holm or Bonferroni procedure, or pre-commit to a small number of comparisons before looking at the results. Testing variants until one crosses p < 0.05 and then reporting only that one is p-hacking, and it produces findings that will not replicate.

Sample size, power, and the limits of small benchmarks

Power — the chance of detecting a real difference — depends on how many items can actually distinguish the two systems. On a 164-item benchmark, a three-point accuracy gap is roughly five items, and the discordant count may be small enough that the test cannot separate a real improvement from noise. This is not a flaw in the test; it is a limit of the evidence.

When you cannot expand the test set, the honest move is to report the confidence interval and say the difference is unresolved. Do not report a non-significant result as proof of equivalence, and do not report a significant result on 20 discordant items as if it were a robust finding. If the decision matters, evaluate on a larger held-out set before committing.

Common failure modes

  • Running a t-test on two aggregate accuracy scores. It discards the pairing and is the wrong model for binary per-item outcomes.
  • Using an unpaired test on paired data. If both systems saw the same items, an unpaired test throws away information and loses power.
  • Ignoring multiple comparisons. Sweeping dozens of configurations and reporting the best p-value inflates false positives.
  • Reporting p without effect size. Statistical significance on a tiny effect is still a tiny effect.
  • Reusing the test set for selection and final reporting. Once a set has guided your choices, it is no longer a clean holdout.
  • Ignoring seed variance. A single run per configuration cannot separate model differences from sampling randomness.

A minimal reproducibility checklist

  1. Fix and record the test set, the item count, and the exact revision of both systems.
  2. Store per-item predictions for both systems, keyed by item identifier.
  3. Choose the test before looking at the result, based on the metric and the pairing.
  4. Compute the discordant counts (or the resampled difference distribution) and report them.
  5. Report the effect size, a confidence interval, and the p-value together.
  6. Apply a multiple-comparison correction if you ran more than one comparison.
  7. Seed every stochastic step and publish the seed.

Where to go next

The fastest improvement most evaluation pipelines can make is to stop comparing averages and start comparing items. If you already log per-item outputs, adding a McNemar test is a few lines of code and a binomial call; if you report composite metrics, a seeded paired bootstrap gives you an interval you can defend.

If you try this on your own evaluation harness — local or hosted — share your setup, the metric, the number of items, and what the test told you. Concrete reports of where a significance test changed a decision, or where it failed to resolve one, are more useful to the community than another leaderboard.

FAQ

Why use McNemar's test instead of a t-test on the two accuracy scores?

A t-test on aggregate scores treats the two averages as independent samples and ignores which items each system got right. McNemar's exact test works on the paired item-level outcomes and analyzes only the items where the systems disagree, which is where the evidence actually lives.

How many discordant pairs do I need for a meaningful result?

There is no universal threshold; power grows with the number of discordant pairs. In practice, report b and c alongside the p-value so readers can judge whether the test had enough disagreements to resolve the question. A non-significant result on very few discordant pairs means unresolved, not equivalent.

Can I compare more than two models with the same approach?

Yes, but pairwise testing across many models inflates the false-positive rate. Either correct with a Holm or Bonferroni procedure, or run an omnibus test first and follow up only on the comparisons that matter. Dror et al. discuss multiple-comparison corrections for NLP evaluation setups.

Does a p-value below 0.05 mean the new model is better in production?

No. It means the observed difference would be unlikely if the two systems were equally good on this test set. Whether that difference matters depends on the effect size, the confidence interval, and whether your test set resembles production traffic.

References

  1. https://aclanthology.org/P18-1128.pdf
  2. https://docs.nvidia.com/nemo/evaluator/tutorials/compare

Related terms

Null hypothesis
The assumption that the two systems are equally good on every item; the test measures how surprising the observed difference is under that assumption.
p-value
The probability of observing a difference at least as large as the one measured, if the null hypothesis were true. It is not the probability that one system is better.
Discordant pair
An item where one system is correct and the other is wrong. Only discordant pairs carry information in McNemar's test.
Effect size
The magnitude of the measured difference in the metric's own units, reported separately from statistical significance.
Paired bootstrap
A resampling procedure that draws test items with replacement many times, recomputes the metric difference each time, and uses the resulting distribution for intervals and p-values.
Multiple comparisons
The inflation of false-positive risk that occurs when many hypotheses are tested on the same data; corrected with procedures such as Holm or Bonferroni.

Leave a Reply

Your email address will not be published. Required fields are marked *