When Zero Findings Means Nothing Was Read
A security scan with zero findings looks the same whether the scanner read the code or not. ai-sec, an AI security assessment bundle for Claude Code, grades a module blind when its scanner could not read the target, and lists every gap under Could not evaluate.
Key takeaways
- A security scan with zero findings means nothing unless the scanner could actually read the target.
- On OWASP Juice Shop, which ships no lockfile, trivy exits 0 with no dependency findings because a version range is not a version.
- ai-sec lets a scanner declare a partial refusal: it still runs, and the one capability it cannot use grades its module blind.
- Every ai-sec run ends with a Could not evaluate block, and a run that completes with a non-empty ledger says so.
- ai-sec scans at T0 by default, sends nothing, and refuses any tier above the one a human wrote into scope.json, with exit code 2.
Contents
FAQ
Does ai-sec need scanners installed?
No. It needs Node 20 or later. Two scanners are built in, builtin for code and probe for a live target; Semgrep, Trivy, Gitleaks, Nuclei and ZAP are used when they are on PATH, and doctor says what each absence costs.
Can I point ai-sec at any website?
No. Only authorized targets: your own code, employer systems with authorization, staging, CTF and deliberately vulnerable practice apps. A host outside the allowlist or a tier above the ceiling in scope.json is refused with exit code 2.
Is ai-sec a pentest?
No. It writes no exploits, business logic is out of reach by design, and nobody outside the project has reviewed it yet.
Sources
Only you and the owner see your answer and rating. Send private feedback
Enjoyed it? Support fx-blog with a coffee.
Related posts
Grade Your Security Scanner Against Answers It Didn't Write
A scanner that scores full marks on its own fixture has proved nothing. A security scanner benchmark built on Juice Shop's own source annotations found 2 of 27 on the first run, and the answer keys broke the report, a rule, the merge and the scorer.
The SQL Injection Sat Below 188 Test-Suite Findings
A security scan report sorted by severity buried a SQL injection under 188 findings from Juice Shop's test suite. How to read an ai-sec report: two axes, a counted non-application block, Could not evaluate, and a grade for every module.
Comments
Oldest firstNo comments yet. Start the conversation.
Here is one row from a scan of OWASP Juice Shop, a deliberately vulnerable practice app:
Two engines installed, both credited with M2 (dependency CVEs), and not one finding. The hurried reading is "nothing was found". The true one is "nothing was read".
This series, in three parts:
Why a clean scan can mean nothing
Juice Shop ships no lockfile. Its
package.jsonstates"express": "^4.17.0", and a range is not a version. trivy will not guess which one is installed, so it has nothing to match against its database. It exits 0 and reports no dependency results, and every layer above it is right to say there were none.That is the usual failure: scanners present an empty result the same way whether the code is clean or the parser crashed.
Five questions to ask of any security scan
How ai-sec works: seven stages, gates in code
ai-sec runs in Claude Code, but the gates are not prompts. A deterministic controller owns them, and the AI explains and judges without sitting in the hot loop.
Two scanners are built in and need nothing installed:
builtinreads code,probereads a live target. Semgrep, Trivy, Gitleaks, Nuclei and ZAP plug into the same adapter protocol when they are on PATH. I chose not to write a detection engine; the value is the layer above one.One real run: the row that stopped lying
I could have refused trivy outright on a target with no lockfile. On Juice Shop that would have thrown away its misconfiguration and secret checks, which found 94 findings, to report one gap. So trivy's preflight now declares a partial refusal: the scanner runs, and the one capability it cannot use is named.
No score went up. The same sentence reaches the report:
Get it, and what it does not do
It needs Node 20+ and nothing else.
mock/is a read-only app with planted defects.scanwith no tier is T0: it reads files and sends nothing.It is not a pentest and writes no exploits. Business logic (M11) is
no-engineon purpose, and nobody outside the project has reviewed it yet.Try it
Read the last block first. On Juice Shop, M2 is now honestly
blind, but that scan also returned 246 findings, and the SQL injection in the login route was not on the first screen. Part 2, The SQL Injection Sat Below 188 Test-Suite Findings, reads that report in the order it was built to be read.Read this with your AI