Probanza — Validation Evidence

An AI system fails in ways ordinary code review does not catch. Probanza reads a codebase and finds those places before they ship.

It is a static auditor: it finds the exposure, it does not stop an attack. Every count below is what it found in real, shipping open-source products — hand-labeled, one finding at a time.

The tables below use the vocabulary of detection measurement. It decodes to four ideas:

A tool can look perfect by flagging almost nothing — high precision, terrible recall — or by flagging everything, which is perfect recall and unusable noise. Both numbers are published here for every rule, which is the only way either one means anything.

Every number is computed from hand-labeled CSVs at build time. None is transcribed by a person, and the build fails if this page drifts from the labels. That is not fastidiousness: prose did drift here once, and it hid a real false positive.

Two rules carry an arrow. They are rules whose false positives we found in our own tool and fixed. The left number is what the rule actually scored when we labeled it; the right is what it scores after the fix.

We publish both, because a product that only ever shows you its final number is asking you to trust that it looked.

• AI_AGENT_SHELL_WITHOUT_APPROVAL — 5 false positives found and eliminated at 2026-07-19b: precision 0.167 → 1.000 over 1 surviving true positives.
• AI_PROMPT_WITHOUT_VALIDATOR — 2 false positives found and eliminated at 2026-07-17a: precision 0.917 → 0.917 over 22 surviving true positives.
• HARDCODED_CREDENTIALS — 53 false positives found and eliminated at 2026-07-12c: precision 0.243 → 1.000 over 17 surviving true positives.
• TEST_GAP_DETECTED — 5 false positives found and eliminated at 2026-07-14x: precision 0.966 → 1.000 over 144 surviving true positives.
• TEST_GAP_DETECTED — 6 false positives found and eliminated at 2026-07-14aa: precision 0.993 → 0.999 over 800 surviving true positives.
• TEST_GAP_DETECTED — 26 false positives found and eliminated at 2026-07-14ae: precision 0.859 → 1.000 over 158 surviving true positives.
• TEST_GAP_DETECTED — 49 false positives found and eliminated at 2026-07-16a: precision 0.942 → 1.000 over 799 surviving true positives.
• TEST_GAP_DETECTED — 27 false positives found and eliminated at 2026-07-17a: precision 0.895 → 0.947 over 229 surviving true positives.
• UNSAFE_EVAL_USAGE — 10 false positives found and eliminated at 2026-07-12c: precision 0.524 → 1.000 over 11 surviving true positives.

A rule can score beautifully on the code its author was staring at while writing it. The only honest test is code the rule has never seen — so five products (browser-use, private-gpt, open-webui, LibreChat, Flowise) were acquired after the same-file rules were built, and those rules met them cold.

What this proves. Run cold against 5 unfamiliar products and 5,592 files, the same-file rules raised 5 false positives — and a manual source-to-prompt census confirmed the silence was correct: those products defend their prompts properly, delimiting untrusted text and carrying it in its own message role. The rules did not overfit to noise. Pointed at unfamiliar, well-built code, they had the good sense to say nothing.

What this does not prove — and we will not pretend otherwise. They also found nothing new there. Zero true positives in the acquired slice makes this a test of specificity, not of discovery. It shows the rules do not invent findings in code they have never read; it does not show they would find real ones.

The cross-file rule does find real things across unfamiliar products — 8 of its 24 true positives live outside the benchmark, at 1.000 precision, so it is plainly not a two-product special case. But it was designed with all twenty products already in view. Its numbers are in-sample by construction, and it has not yet had a true out-of-sample test. Claiming one would be the cheapest lie in this document.

Across every rule, the acquired slice contributes 45 true positives and 5 false positives.

The credential rule detects credentials. Whether a credential is an incident is a second question — and answering only the first is how security tools earn their reputation for noise.

A PostHog project key is a hardcoded credential. The vendor also publishes it on purpose, to be embedded in client code. Calling that CRITICAL is not a detection error; it is a risk-classification error. So the rule keeps detecting every credential, and a taxonomy decides what kind it is.

What they turned out to be:

• PostHog Project API Key × 11
• Default Development Credential × 3
• Development Credential × 2
• Mixpanel Project Token × 1

Two properties keep the taxonomy honest. A recognized secret format beats context: an AWS key inside a CI workflow is still a leaked AWS key, and a dev-looking location never rescues real secret material. And ignorance never downgrades: a credential the taxonomy does not recognize stays CRITICAL as unclassified, which makes that count a quality metric rather than a hiding place.

A security tool's real price is not its licence. It is the hours an engineer spends triaging what it says — and a tool that shouts about everything teaches its users to hear nothing.

Two arcs did this without weakening detection by a single finding: one removed the rules' false positives, the other stopped calling public-by-design credentials incidents. A team triaging the security rules across these twenty products would have opened 122 alerts. Today they open 42 — 66% fewer, and every true positive is still found.

This is the section most detection products cannot write, because writing it requires having measured something and then thrown it away. Each of these was a plausible improvement that a reasonable engineer would have shipped on intuition.

The cross-file injection rule's true positives are real sites in shipping open-source products: external retrieved content — scraped pages, fetched documents, RAG results — reaching a prompt instruction with no instruction/data boundary.

These are architectural observations, not vulnerability reports. For a research agent, blending scraped text into a prompt is the product; the finding names the boundary that is missing and the exposure that follows from its absence. We make no exploitability claim, ship no proof-of-concept, and have not filed these as security issues.

They are here because they reproduce: pin the commit, run the auditor, get the same findings.

The access-control rule is a different matter, and we treat it as one. A missing authorization check on an object-id handler is a vulnerability class (IDOR), not an architectural style — so the number on the table above is reported as a measurement of the rule, over a corpus adjudicated for precision, and nothing here names a file, a line, or a route in any product. We have not assessed exploitability against any deployment and have not notified maintainers, so this page does not identify the handlers. What the rule can see is the code-shaped half only: it reads the absence of a check in a repository, never a live deployment — the other half of the breach that motivated the rule (a real product's admin account with the password 123456) is invisible to any static scan, and the page says so plainly under What it cannot see.

Every detection vendor publishes what it catches. Here is one it does not — the real code, the real reason, and the real attempt to fix it.

A boundary a tool states is worth more than a capability it claims, because only one of the two can be checked.

That is the shape of every boundary here. Not a gap we have not gotten to — a trade made with the numbers in front of us: one known miss beats one unknown false alarm, because the miss is registered and the false alarm is what teaches your team to stop reading the output.

Where the auditor cannot see, a finding's absence means nothing. That list is maintained as a first-class artifact, not an afterthought:

• Languages — TypeScript/JavaScript and Python only. Go, Java, Ruby and Rust are unsupported.
• Notebooks (.ipynb) are scanned but carry no labeled rows. Unmeasured.
• Static analysis only — no code is executed. Dynamic imports and framework-specific runtime wiring can escape the uses graph. Every audit report carries this in its uncertainty block.
• Registered false negatives are listed per rule, each with the arc that measured it.

Not synthetic fixtures — shipping code with real users, chosen because it contains the surfaces the rules target: model calls, prompt templates, retrieval pipelines, agent tool-use.

Benchmark slice (15) — the labeled ledger's lineage: aider, files-to-prompt, gpt-researcher, instructor, langfuse, mem0, MetaGPT, openai-node, openai-python, openai-structured, prompt-cleaner-mcp, prompt-file-examples, Prompt-Injection-Playground, screenshot-to-code, vercel-ai.

Not every label behind this page is downloadable, and we would rather state the split than let you discover it.

The public tier — the commodity detections: credentials, unsafe evaluation, shell execution, and the model-call validator family — ships its full ground truth in this repository: 1345 labeled findings, every CSV beside the dated record that produced it, recomputable by verify.py.

The gated tier — the differentiated detectors: the injection family, access control, the test-gap and agent-shell rules — publishes its numbers, its boundaries and its registered misses on this page, and ships its ground truth — 2478 labeled findings — as an evaluator bundle, under NDA, on request. The bundle carries the labels, the full dated records, and its own copy of the verify script that recomputes every gated number the same way.

The reason is stated plainly rather than buried: the labeled corpus is the asset a copy of this product would need most. Handing the commodity part to everyone costs nothing; handing the differentiated part to anonymous readers is a disclosure decision, and we made it the way this project makes every decision — recorded, with the trade named (decision 004). An evaluator with a name gets everything.

A measurement document that asks to be believed has failed. So here is everything needed to disbelieve this one.

The public tier's labeled data is in this repository. Every finding behind the public rows above — 1345 of them, each marked true positive, false positive or false negative by hand — is committed as CSV beside the dated record that produced it.

Recompute the numbers without taking our word for anything. A dependency-free script reads those CSVs and prints their table rows from scratch:

If its output disagrees with what you read above, the page is wrong and you should say so.

The gated tier is checkable the same way — by named evaluators. The bundle carries the remaining 2478 labeled findings and its own verify script. A number you cannot recompute from this repository is marked bundle in the table, never smuggled past you.

Every public-tier claim links to the dated record that grounds it; gated rows name their arc. The corpus is pinned to exact commits, so the code we audited is the code you would fetch.

Probanza's detection engine is commercial and its source is closed, so no, you cannot run it from this repository. That is the trade — stated in one sentence, rather than buried in a footnote.

What we will not do is ask you to take the rest on faith. The commodity labels, the public records, the pinned corpus, the doctrines we measured and threw away, and the failures we cannot see are all here — and everything gated is one named request away, checkable by the same script.

A vendor confident in its product can afford to show its homework — and to say which pages of it are licensed reading. That is the argument of this page, and it is the only one we intend to make.