Skip to content
fx-blog

Grade Your Security Scanner Against Answers It Didn't Write

A scanner that scores full marks on its own fixture has proved nothing. A security scanner benchmark built on Juice Shop's own source annotations found 2 of 27 on the first run, and the answer keys broke the report, a rule, the merge and the scorer.

Harry4 min readAdvanced
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A benchmark you wrote can only award self-measured; ai-sec grades a module measured only on a corpus nobody on the project wrote.
  • OWASP Juice Shop annotates each planted defect in its own source, and benchmarks/juiceshop.mjs re-derives the answer key from a clone.
  • The first ai-sec run against Juice Shop found 2 of 27 annotated lines, and eight rules written from the misses raised it to 16 of 27.
  • The benchmarks exposed a report that was 76% test suite, a rule naming the wrong cause, a merge that discarded findings and a scorer that never read occurrences.
  • Defects a scanner cannot find stay in the answer key with a written reason, and the benchmark test fails if more than half the key is out of reach.
Contents

TL;DR A fixture you wrote can only agree with you, so a scanner that scores full marks on it has proved nothing. A security scanner benchmark is worth something when someone else wrote the target, and better still, the answers. Only here: ai-sec re-derives its OWASP Juice Shop answer key from Juice Shop's own source annotations, and the first run against it found 2 of 27.

Do this: Clone OWASP NodeGoat, copy in ai-sec's committed answer key, scan it read-only at T0, and run sec modules.

Part 2, The SQL Injection Sat Below 188 Test-Suite Findings, ended on nine modules graded measured on Juice Shop: full marks on a corpus nobody here wrote. This part opens the answer keys behind that grade, and the first run before any rule was added.

mock/, the app I wrote with planted defects, gives every module full marks. OWASP NodeGoat gave three modules zero. Nothing in the ruleset changed between those two runs. Only the target did.

This series, in three parts:

  1. When Zero Findings Means Nothing Was Read
  2. The SQL Injection Sat Below 188 Test-Suite Findings
  3. Grade Your Security Scanner Against Answers It Didn't Write (you are here)

Four benchmarks, because they answer different questions

target written by answers written by evidence best grade
mock/ us us reading files self-measured
OWASP NodeGoat someone else us, from its tutorials reading files measured
OWASP Juice Shop someone else the target itself reading files measured
benchmarks/dynamic/ us us sending requests self-measured

Juice Shop writes its own answers

Juice Shop marks the exact line of each planted defect in its own source:

tsJuice Shop source, as quoted in benchmarks/README.md
models.sequelize.query(`SELECT ... '${req.body.email}' ...`) // vuln-code-snippet vuln-line loginAdminChallenge

benchmarks/juiceshop.mjs reads the 27 markers back out of a clone, so the key is re-derived, not trusted. On NodeGoat I chose which defects to record. Here I chose nothing.

The first run found 2 of 27, both SQL injection, both a shape mock/ already had. The misses were the lesson: Angular's bypassSecurityTrustHtml, an allowlist checked with url.includes, a guard commented out one character from live, a coupon tool whose limit lived only in a sentence to the model. Eight rules later, each written to generalise past this corpus: 16 of 27.

The showcase grades page: seven grades from measured to no-engine, and sec modules on Juice Shop showing M2 blind, M4 and M7 measured, M11 no-engine, 3 partial, 1 blind, 9 measured, 1 no-engine

What the answer keys broke

What broke What showed it
The report was 76% test suite 188 of 246 findings in test/ (part 2)
A rule named the wrong cause nosql-operator-injection fired on a Sequelize where:, which binds the value. The real defect was whose wallet gets credited
The merge discarded findings the dynamic benchmark reported 4 of 7 header defects; probe had found all 7, and a shared CWE family folded three away
occurrences was never scored a finding anchored at the cause, listing the right call site, counted as a miss
Any expected.json was an answer key a repo with its own golden file would have been graded against somebody else's data

Questions it cannot answer stay in

Three Juice Shop defects are still unfound, and I kept them in the denominator:

  • unguarded frontend routes: a route without a guard is the normal case;
  • crypto addresses in the redirect allowlist: a business question;
  • a review update that ignores the owner: the variable is used, just not for authorization.

Deleting the questions you cannot answer turns a benchmark into a decoration, so the benchmark test fails if more than half the key is out of reach. Two answers that moved to "found" after the rules were written carry a was_missed_until field saying why.

A target that is meant to be attacked

The dynamic benchmark starts a vulnerable server I wrote on a loopback port, scans it at T2 and stops it. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its own scope.json.

mermaid
flowchart LR
  A[as the owner<br/>what the object looks like] --> B[with no session<br/>the control]
  B --> C[as the other account<br/>the test]
  B -- object still returned --> P[public: a different defect]
  C -- same body digest --> I[cross-account read]

Two endpoints are correct on purpose. /api/inbox answers 200 to every account with that account's own data, so a check that reads only the status code reports it falsely.

Try it

NodeGoat is a deliberately vulnerable practice app. It is scanned read-only at T0 and never run.

Try it
git clone --depth 1 https://github.com/OWASP/NodeGoat.git /tmp/nodegoat
cp benchmarks/nodegoat-expected.json /tmp/nodegoat/expected.json
node bin/sec.mjs scan /tmp/nodegoat --slug ng
node bin/sec.mjs modules --slug ng

Without trivy on PATH, M2 and M10 grade failing there, and the ledger says why.

Read this with your AI

textPaste into your AI
Read the post "Grade Your Security Scanner Against Answers It Didn't Write".
1. Summarise it in 5 bullets for an engineer who owns a security scanning setup.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Name one public practice app I could use as an answer key for a scanner I run.

FAQ

Why scan OWASP Juice Shop and NodeGoat?

They are deliberately vulnerable practice applications that someone else wrote, so they can surprise the scanner. ai-sec scans a clone read-only at T0 and never runs them.

Does the dynamic benchmark attack anything?

Only a server ai-sec starts itself from its own repository. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its scope.

Sources

  1. ai-sec benchmarks README
  2. ai-sec evidence: the dynamic benchmark (M15)
  3. OWASP NodeGoat
  4. OWASP Juice Shop

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

When Zero Findings Means Nothing Was Read

A security scan with zero findings looks the same whether the scanner read the code or not. ai-sec, an AI security assessment bundle for Claude Code, grades a module blind when its scanner could not read the target, and lists every gap under Could not evaluate.

4 min readBeginner

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000