The SQL Injection Sat Below 188 Test-Suite Findings
A security scan report sorted by severity buried a SQL injection under 188 findings from Juice Shop's test suite. How to read an ai-sec report: two axes, a counted non-application block, Could not evaluate, and a grade for every module.
Key takeaways
- Scanning OWASP Juice Shop produced 246 findings, 188 of them hardcoded credentials in the test suite, and severity order put seed passwords above the SQL injection in the login route.
- An ai-sec report has four parts: the verdict, the findings, the non-application block and Could not evaluate.
- Every finding has a severity, how bad it is, and a confidence, how sure, and Certain needs a reproduction by someone other than the tool that found it.
- Findings in test, fixture, example, generated or vendored code are capped at Tentative, grouped and counted, and never fail a build on their own.
- On Juice Shop, sec modules grades 9 modules measured, 3 partial, 1 blind and 1 no-engine, and a grade should be quoted with the module.
Contents
FAQ
Why not just drop findings in test files?
A credential in a test file can still be a live credential. ai-sec reports and counts those findings in one block and keeps them out of the verdict, so nothing is deleted.
What does exit code 4 mean in CI?
The run was degraded: something ran and something did not. It is not a pass; you did not get the scan you asked for.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Grade Your Security Scanner Against Answers It Didn't Write
A scanner that scores full marks on its own fixture has proved nothing. A security scanner benchmark built on Juice Shop's own source annotations found 2 of 27 on the first run, and the answer keys broke the report, a rule, the merge and the scorer.
When Zero Findings Means Nothing Was Read
A security scan with zero findings looks the same whether the scanner read the code or not. ai-sec, an AI security assessment bundle for Claude Code, grades a module blind when its scanner could not read the target, and lists every gap under Could not evaluate.
Comments
Oldest firstNo comments yet. Start the conversation.
Part 1, When Zero Findings Means Nothing Was Read, ended with OWASP Juice Shop's M2 row honestly
blind. The same scan returned 246 findings, and the SQL injection in the login route was not on the first screen. This part reads that report the way it was built to be read.The scan of Juice Shop, read-only at T0, produced 246 findings. 188 of them were hardcoded credentials in
test/. BecauseCriticalsorts aboveHigh, the first three screens weredata/static/users.ymlseed passwords andtest/api/2fa.test.ts.This series, in three parts:
Four parts, in this order on purpose
FAIL — 3 finding(s) at or above the bar (High).Could not evaluate: every gap, and how to close it.Severity says how bad. Confidence says how sure
High / TentativeandHigh / Certainare different jobs: the first needs someone to look, the second needs someone to fix. Severity is rescored in context, never copied from the tool, and when two scanners disagree the report says so.Findings from several scanners merge by CWE family, not rule id, so
open-redirectandjavascript.express.security.audit.express-open-redirectare one finding with two sources.The test suite is reported, not counted
Every one of those 188 findings really was in that file. The report was accurate and nobody would have read it, which is worse than a false positive: a false positive gets argued with.
So every finding is now labelled
application,test,fixture,example,generatedorvendored, and the rest are grouped:Nothing is dropped. A credential in a test file can be a live one, and a rule that deleted it would be the leak. When a path matches no known shape, I made it classify as
application: calling a handler a test hides a real finding, while calling a test a handler only costs attention.Then ask what each module's zero is worth
sec modulesgrades all fourteen modules against a benchmark with recorded answers. On Juice Shop:Quote the grade with the module. "M5 found nothing" means something different when M5 is
measuredand when it isunmeasured. M11, business logic, staysno-engineon every run: an automated M11 that finds nothing would be worse than saying so.Twelve layers of thirty-two
The coverage map lists thirty-two security layers and who owns each. Twelve are covered and graded for the product being scanned. ZAP and nuclei are built but blocked on this machine. Privacy and PII data-flow is covered by nothing, and that is the gap most likely to matter legally.
Try it
Treat 4 as a warning you investigate, not a pass. On Juice Shop, nine modules read
measured: full marks on a corpus nobody here wrote. Whose answers were those, and what did the first run score before any rule was added? Part 3, Grade Your Security Scanner Against Answers It Didn't Write, opens the answer keys.Read this with your AI