Grade Your Security Scanner Against Answers It Didn't Write
A scanner that scores full marks on its own fixture has proved nothing. A security scanner benchmark built on Juice Shop's own source annotations found 2 of 27 on the first run, and the answer keys broke the report, a rule, the merge and the scorer.
Key takeaways
- A benchmark you wrote can only award self-measured; ai-sec grades a module measured only on a corpus nobody on the project wrote.
- OWASP Juice Shop annotates each planted defect in its own source, and benchmarks/juiceshop.mjs re-derives the answer key from a clone.
- The first ai-sec run against Juice Shop found 2 of 27 annotated lines, and eight rules written from the misses raised it to 16 of 27.
- The benchmarks exposed a report that was 76% test suite, a rule naming the wrong cause, a merge that discarded findings and a scorer that never read occurrences.
- Defects a scanner cannot find stay in the answer key with a written reason, and the benchmark test fails if more than half the key is out of reach.
Contents
FAQ
Why scan OWASP Juice Shop and NodeGoat?
They are deliberately vulnerable practice applications that someone else wrote, so they can surprise the scanner. ai-sec scans a clone read-only at T0 and never runs them.
Does the dynamic benchmark attack anything?
Only a server ai-sec starts itself from its own repository. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its scope.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
The SQL Injection Sat Below 188 Test-Suite Findings
A security scan report sorted by severity buried a SQL injection under 188 findings from Juice Shop's test suite. How to read an ai-sec report: two axes, a counted non-application block, Could not evaluate, and a grade for every module.
When Zero Findings Means Nothing Was Read
A security scan with zero findings looks the same whether the scanner read the code or not. ai-sec, an AI security assessment bundle for Claude Code, grades a module blind when its scanner could not read the target, and lists every gap under Could not evaluate.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 2, The SQL Injection Sat Below 188 Test-Suite Findings, ended on nine modules graded
measuredon Juice Shop: full marks on a corpus nobody here wrote. This part opens the answer keys behind that grade, and the first run before any rule was added.mock/, the app I wrote with planted defects, gives every module full marks. OWASP NodeGoat gave three modules zero. Nothing in the ruleset changed between those two runs. Only the target did.This series, in three parts:
Four benchmarks, because they answer different questions
mock/self-measuredmeasuredmeasuredbenchmarks/dynamic/self-measuredJuice Shop writes its own answers
Juice Shop marks the exact line of each planted defect in its own source:
benchmarks/juiceshop.mjsreads the 27 markers back out of a clone, so the key is re-derived, not trusted. On NodeGoat I chose which defects to record. Here I chose nothing.The first run found 2 of 27, both SQL injection, both a shape
mock/already had. The misses were the lesson: Angular'sbypassSecurityTrustHtml, an allowlist checked withurl.includes, a guard commented out one character from live, a coupon tool whose limit lived only in a sentence to the model. Eight rules later, each written to generalise past this corpus: 16 of 27.What the answer keys broke
test/(part 2)nosql-operator-injectionfired on a Sequelizewhere:, which binds the value. The real defect was whose wallet gets creditedprobehad found all 7, and a shared CWE family folded three awayoccurrenceswas never scoredexpected.jsonwas an answer keyQuestions it cannot answer stay in
Three Juice Shop defects are still unfound, and I kept them in the denominator:
Deleting the questions you cannot answer turns a benchmark into a decoration, so the benchmark test fails if more than half the key is out of reach. Two answers that moved to "found" after the rules were written carry a
was_missed_untilfield saying why.A target that is meant to be attacked
The dynamic benchmark starts a vulnerable server I wrote on a loopback port, scans it at T2 and stops it. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its own
scope.json.Two endpoints are correct on purpose.
/api/inboxanswers 200 to every account with that account's own data, so a check that reads only the status code reports it falsely.Try it
NodeGoat is a deliberately vulnerable practice app. It is scanned read-only at T0 and never run.
Without
trivyon PATH, M2 and M10 gradefailingthere, and the ledger says why.Read this with your AI