Articles

To benchmark a security linter or SAST tool, you need labeled examples: 'this code is vulnerable,' 'this code is safe.' That labeling decision IS the ground truth. It's made by a human, not an algorithm — and the benchmark inherits every assumption that human brought.

CWE-89 means SQL injection. That's a name, not a verdict. What the Common Weakness Enumeration actually encodes — views, abstraction levels, the Top 25 formula — and why a CWE id can never carry a severity.

A scanner can print CVSS 9.1 and the word LOW on the same line. What the score actually measures, what it doesn't, and why you should gate on the number — never the word.

An index looks like a measurement of 'the market.' It's a composite — a committee picks the members, a rule sets the weights, a calendar sets the rebalance. Editorial choices dressed as a number. Every overall score works this way; the honest ones publish the weights, the rationale, and what happens when the weights move.

A cancer screen can be 90% accurate and still wave through two-thirds of the tumors it exists to catch. The confusion matrix is the honest ledger behind that number — and accuracy is usually the last thing you should read.

Two radiologists agreed on 92 of 100 scans. Cohen's kappa scores that at 0.16 — barely above chance. When labels are judgment calls, raw percent agreement is the wrong check. Here is the right one.

Every benchmark on this site compares my own ESLint plugins to competitors — a real conflict of interest, not a disclaimer I can write my way out of. Here's the actual, checkable process I use instead of asking you to trust my verdict.

When a measure becomes a target, it ceases to be a good measure. Goodhart's Law is the failure mode where moving a number and improving the thing the number stood for split into two projects — and the cheaper one wins by default.

eslint-plugin-jwt gets scored on precision and recall against a labeled vulnerability corpus. My serverless caching plugin doesn't — and 3 of its 4 benchmark suites aren't built yet. Here's why that gap is honest, not an inconsistency.

A blood panel, a fund's track record, and a security benchmark can all report a clean, confident number that is quietly wrong. Measurement bias doesn't announce itself — the five kinds, and how to catch each before you publish.

A test that scores 95% precision on a balanced sample can be right under 2% of the time when the thing it hunts is rare. That's not a bug in the arithmetic — it's the base rate problem, and it's why benchmark conditions rarely predict field behavior.

A CVSS 9.1 SSRF rule ships labeled LOW. A CVSS 9.8 JWT rule ships labeled MEDIUM. The root cause was one unvalidated string field.