Skip to main content
ofri peretz

The Foundations Series

Every benchmark, severity score, and leaderboard ends in a number someone wants you to trust. This series walks the whole chain: what the counts are, what they summarize, why context changes them, how measurement lies, who wrote the answer key, and how numbers turn into verdicts. Read it in order — each chapter ends with the question the next one answers.

  1. Confusion matrixfoundations

    Four counts sit under every detection claim. If you can't name them, you can't argue with the metric.

  2. What the four counts collapse into — and what each summary quietly throws away.

  3. A 95%-accurate scanner that's usually wrong when it fires. Prevalence does that, and nobody prints it.

  4. How a measurement lies while everyone involved is being honest.

  5. Goodhart's Lawfoundations

    Point a target at a number and watch the number stop meaning anything.

  6. Same data, same result is table stakes. New data, same claim is where trust starts.

  7. A benchmark too small to detect the difference it reports isn't evidence. How many cases is enough.

  8. What a p-value actually says — which is less than you think, and still worth having.

  9. A leaderboard is an ordering, not a measurement. First place says nothing about the gap.

  10. Two raters agree 80% of the time — chance alone gets you most of the way there. κ subtracts the freebie.

  11. A metric can give the same wrong answer every time. Consistent is not correct.

  12. Proxy metricsfoundations

    Downloads, stars, views — you're not measuring the thing, you're measuring its shadow.

  13. One score means someone chose the weights. Find them before you trust the total.

  14. Ground truthvocabulary

    Every benchmark has an answer key, and a person wrote it. “Correct” is a decision before it's a fact.

  15. CVSS scoresvocabulary

    What's inside a severity score — and why a 9.1 can still be the wrong thing to fix first.

  16. CWE taxonomyvocabulary

    The weakness tree every scanner cites. Counting IDs without the hierarchy compares parents to their own children.

  17. OWASP Top 10vocabulary

    An awareness document, not a checklist — the difference changes what “covered” is allowed to mean.

  18. Two ways a tool decides code is dangerous: follow the data, or match the shape. They fail differently.

  19. Three names for reading code without running it. Where the categories end and the marketing begins.

foundations = the statistics of measurement itself. vocabulary = the security and static-analysis terms the measurements are about. Both are written to be timeless — no tool versions, no news pegs.