The Foundations Series
Every benchmark, severity score, and leaderboard ends in a number someone wants you to trust. This series walks the whole chain: what the counts are, what they summarize, why context changes them, how measurement lies, who wrote the answer key, and how numbers turn into verdicts. Read it in order — each chapter ends with the question the next one answers.
- Confusion matrixfoundations
Four counts sit under every detection claim. If you can't name them, you can't argue with the metric.
- Precision, recall, F1vocabulary
What the four counts collapse into — and what each summary quietly throws away.
- Base rate problemfoundations
A 95%-accurate scanner that's usually wrong when it fires. Prevalence does that, and nobody prints it.
- Bias in measurementfoundations
How a measurement lies while everyone involved is being honest.
- Goodhart's Lawfoundations
Point a target at a number and watch the number stop meaning anything.
- Reproducibility vs replicabilityfoundations
Same data, same result is table stakes. New data, same claim is where trust starts.
- Sample size & powerfoundations
A benchmark too small to detect the difference it reports isn't evidence. How many cases is enough.
- p-values & significancefoundations
What a p-value actually says — which is less than you think, and still worth having.
- Ranking vs measuringfoundations
A leaderboard is an ordering, not a measurement. First place says nothing about the gap.
- Cohen's κ & inter-rater agreementfoundations
Two raters agree 80% of the time — chance alone gets you most of the way there. κ subtracts the freebie.
- Valid vs reliable metricsfoundations
A metric can give the same wrong answer every time. Consistent is not correct.
- Proxy metricsfoundations
Downloads, stars, views — you're not measuring the thing, you're measuring its shadow.
- Composite scores & weightingfoundations
One score means someone chose the weights. Find them before you trust the total.
- Ground truthvocabulary
Every benchmark has an answer key, and a person wrote it. “Correct” is a decision before it's a fact.
- CVSS scoresvocabulary
What's inside a severity score — and why a 9.1 can still be the wrong thing to fix first.
- CWE taxonomyvocabulary
The weakness tree every scanner cites. Counting IDs without the hierarchy compares parents to their own children.
- OWASP Top 10vocabulary
An awareness document, not a checklist — the difference changes what “covered” is allowed to mean.
- Taint vs heuristic detectionvocabulary
Two ways a tool decides code is dangerous: follow the data, or match the shape. They fail differently.
- Static analysis vs SAST vs lintingvocabulary
Three names for reading code without running it. Where the categories end and the marketing begins.