ofri peretzA fraud filter that catches zero of the last 200 confirmed-fraudulent charges needs no p-value to convict it. A trading strategy that beat its benchmark by five points last quarter might be genuine skill — or the kind of luck that evaporates by spring. Same data, same afternoon of analysis, two completely different standards of proof.
A test that returns zero detections across every known case of the thing it claims to catch doesn't need a significance test; that zero is a structural fact, and no amount of re-sampling changes it. But ask a subtler question — is a five-point edge real, or is it sampling noise — and the honest answer depends on numbers a summary table never shows: effect size, variance, statistical power. Knowing which of those two questions you're asking is the whole article.
The first question: does this detector catch the failure at all?
A screening test that returns zero hits across every confirmed case of a disease doesn't need a significance test — you just need a sample large enough that one hit was structurally possible, and even a handful of cases clears that floor. Add forty more known-positive cases, still get zero, and the conclusion doesn't move; the same zero just grows more confident.
The second question is different in kind: is the gap between two contenders large enough to be real? (A composite score like an F1 rests on the same true-positive / false-positive bookkeeping covered in the confusion matrix article.) A five-point difference on a small sample could be genuine — or a fluctuation that vanishes if you swap three cases or re-run on a slightly different set. You cannot tell which without knowing the effect size.
The distinction underneath both: not every result carries sampling variability. Whether a deterministic test fires on a fixed, known input is a fact, not a draw from a distribution — no noise in that outcome to test. Statistical tests exist for questions where repeated sampling would produce a spread of answers. Run the same fixed case through the same check twice and you get the same output; a p-value on that has nothing left to measure.
Statistical power is the probability that a test detects a real effect when one genuinely exists. If the true gap between two contenders is fifteen points and you test it on a sample of forty, how often does the test come back significant? That probability is the power — the field's standard target is 80%, meaning you will still miss one in five real effects. Fine for an exploratory look; too low for a decision you are about to ship.
Three things move it: more observations sharpen the test's sensitivity to smaller effects; larger true effects are easier to detect than small ones; and a stricter significance threshold (α = 0.01 instead of 0.05) cuts false positives but demands a bigger sample to compensate.
Cohen (1988) gives the standard effect-size conventions — small = 0.2, medium = 0.5, large = 0.8 (Cohen's d). At 80% power and α = 0.05, detecting a medium effect needs roughly 64 observations per group; a small effect needs roughly 400.
I learned this at a poker table before I ever met it in a textbook: a player up after forty hands has proven nothing. Variance alone can float a mediocre player for an evening and sink a strong one. The edge — the thing you actually want to measure — only surfaces over thousands of hands. There, sample size isn't a technicality; it's the whole difference between reading skill and reading a good night. n is hands, and forty hands is a story, not a measurement.
The same arithmetic decides which comparisons a report can honestly make. Two diagnostic tests scoring 95% and 45% differ by fifty points — so large any sample of forty detects it. Two tests eight points apart — 63% against 55% — sit close enough that forty cases struggle to separate them from noise without a power calculation first. This is why a careful report leads with the categorical claim ("misses the entire category") and refuses to rank near-neighbours by a handful of points.
The opposite situation looks similar on the surface and needs the opposite treatment. Suppose you A/B-test three versions of a checkout page — call their conversion rates 68%, 71%, and 73% over a few hundred sessions each. The eye wants to crown the 73%. Whether the rate truly differs between versions needs a chi-squared test on the counts; run it and it comes back with a small statistic and p ≈ 0.4 — no significant difference, and the three-way ranking your eye drew is noise.
That test earns its p-value precisely because each visitor is a genuine random draw — the sampling story a hand-built checklist never has. Statistical Significance and p-Values owns the full contingency-table walkthrough, including the paired McNemar's-test version for comparing two tools on the same cases.
This is where most people trip: a confidence interval requires the test set to be a random sample from some population. A hand-built set of cases is not one.
Think of a driving examiner's checklist of manoeuvres, a medical board's curated question bank, or a chess coach's hand-picked tactics puzzles. Each is deliberately constructed to cover chosen categories — which bounds a specific kind of validity (the set covers the categories you chose) without giving you a random draw from the space of situations that show up in the real world. Conflating the two manufactures false precision.
A confidence interval on "score over these forty cases" is computable and practically meaningless: the formula assumes repeated draws from a distribution, and deliberate test cases aren't that. The same problem sinks a p-value on "is performance significantly above 50% here" — it tells you whether you beat a coin flip on these particular cases, not on the next real one you face, because the two don't share a distribution.
What a constructed set is legitimately good for: exact, observational counts — did it catch this case, did it miss an entire category. What it cannot support: a confidence interval or significance claim about performance beyond the cases in it. The table below draws that line for every question type here.
Four rules for reading any benchmark, audit, or scorecard:
1. Category absence is evidence without statistics. Zero out of N on a whole category needs no t-test — the tool has no coverage of that class of problem, full stop.
2. Only run significance tests on a genuine sample. A deliberately built set can't support a p-value about real-world generalization; outcomes drawn or generated at random, like the three-variant test above, can. Know which situation you're in before you reach for the test.
3. State your n before your conclusion. "Forty cases, missed the entire category" is complete on its own. "Forty cases, 5% higher score" needs a power calculation before it means anything — the sample size is part of the finding, not a footnote to it.
4. Report null results. "No significant difference" is an honest finding, not a result to bury. Publishing only the comparisons that reached significance is publication bias — the same distortion covered in Bias in Measurement, operating at the analysis stage instead of during data collection.
Named misconception: "A larger sample always gives more accurate results." A larger sample buys more power to detect small effects. It does not turn a deliberately constructed set into a random sample, and no sample size fixes that. Sample size and sampling method are different problems, and conflating them is what manufactures false precision.
| Question type | Needs significance test? | Why |
|---|---|---|
| Does the tool detect this failure at all? | No | Presence/absence, not magnitude — zero is zero |
| Which of two tools scores higher? | Depends on effect size | Run a power calculation before claiming the difference is real |
| Do three groups have different rates? | Yes (random sample) | Random draws + categorical comparison — chi-squared appropriate |
| Is our score significantly above 90% on a fixed test set? | No (constructed set) | Not a random sample; a CI would be false precision |
| Did it miss an entire category? | No | Observational count — report the zeros directly |
| Is a 5-point improvement real? | Yes | Small effect; small n has low power; significance test or power calc required |
If this is the kind of page you'll want open mid-argument, bookmark it — and follow me on Dev.to to catch the next one in the series.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates. The foundational book on power analysis — defines the small/medium/large effect size conventions (d = 0.2 / 0.5 / 0.8) used throughout statistics and cited in this article.
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. Accessible practical guide to effect size and statistical power — freely available online and the clearest non-textbook treatment of the topic.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. The paper behind the G*Power tool (freely available at gpower.hhu.de) — the standard software for the power calculations referenced in this article.
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLOS Biology, 13(3). On what goes wrong when significance tests are applied carelessly — the publication bias and null-result-suppression problem that motivates rule four above.
Builds on — The Confusion Matrix: the four counts every score in this article is ultimately built from.
Read next — Statistical Significance and p-Values: what a p-value means, the full chi-squared walkthrough, and the paired McNemar's test for comparing two tools on one set.
Cited by — The FP-Tax Benchmark and We Ranked 5 AI Models by Security — the evidence articles whose worked numbers (the leaderboard gaps, the null-result chi-squared) live where the product proof belongs. See also Bias in Measurement for the publication-bias angle.
Foundations series: ← Reproducibility vs replicability · hub · p-values & significance →
Part of the Interlace ESLint ecosystem. Source on GitHub · npm · Follow: Dev.to/ofri-peretz · ofriperetz.dev