Skip to content
fx-blog

When Zero Findings Means Nothing Was Read

A security scan with zero findings looks the same whether the scanner read the code or not. ai-sec, an AI security assessment bundle for Claude Code, grades a module blind when its scanner could not read the target, and lists every gap under Could not evaluate.

Harry4 min readBeginner
  • Judgment
  • Verification
MarkdownSave

Key takeaways

  • A security scan with zero findings means nothing unless the scanner could actually read the target.
  • On OWASP Juice Shop, which ships no lockfile, trivy exits 0 with no dependency findings because a version range is not a version.
  • ai-sec lets a scanner declare a partial refusal: it still runs, and the one capability it cannot use grades its module blind.
  • Every ai-sec run ends with a Could not evaluate block, and a run that completes with a non-empty ledger says so.
  • ai-sec scans at T0 by default, sends nothing, and refuses any tier above the one a human wrote into scope.json, with exit code 2.
Contents

TL;DR A security scan that comes back with zero findings looks the same whether the scanner read your code or never got the chance. ai-sec is an AI security assessment bundle for Claude Code that pairs every result with what it could not check. Only here: a scanner that ran but could not read, like trivy on a repo with no lockfile, grades its module blind instead of clean, and the run says so under Could not evaluate.

Do this: Run node bin/sec.mjs doctor and a T0 scan on a repo you own, and read the Could not evaluate block before the findings.

Here is one row from a scan of OWASP Juice Shop, a deliberately vulnerable practice app:

textsec modules on OWASP Juice Shop, before the fix
  M2            2/2        0      0         0            0  unmeasured

Two engines installed, both credited with M2 (dependency CVEs), and not one finding. The hurried reading is "nothing was found". The true one is "nothing was read".

This series, in three parts:

  1. When Zero Findings Means Nothing Was Read (you are here)
  2. The SQL Injection Sat Below 188 Test-Suite Findings
  3. Grade Your Security Scanner Against Answers It Didn't Write

Why a clean scan can mean nothing

Juice Shop ships no lockfile. Its package.json states "express": "^4.17.0", and a range is not a version. trivy will not guess which one is installed, so it has nothing to match against its database. It exits 0 and reports no dependency results, and every layer above it is right to say there were none.

That is the usual failure: scanners present an empty result the same way whether the code is clean or the parser crashed.

Five questions to ask of any security scan

  1. Did every scanner run, and could it read this target? A missing or half-blind scanner must show up, not vanish.
  2. How sure is each finding? Severity says how bad. You also need how sure.
  3. Is it the application, or the test suite?
  4. Who graded the scanner, and against whose answers?
  5. Who said you may touch this target?

How ai-sec works: seven stages, gates in code

ai-sec runs in Claude Code, but the gates are not prompts. A deterministic controller owns them, and the AI explains and judges without sitting in the hot loop.

mermaid
flowchart LR
  A[discover<br/>map the surface] --> B[scope<br/>tier, hosts, authorization]
  B --> C[scan<br/>T0 sends nothing]
  C --> D[merge<br/>by CWE family]
  D --> E[verify<br/>not the finder]
  E --> F[score<br/>graded on answers]
  F --> G[report<br/>verdict + Could not evaluate]
  B -. tier above ceiling .-> X[REFUSED, exit 2]

Two scanners are built in and need nothing installed: builtin reads code, probe reads a live target. Semgrep, Trivy, Gitleaks, Nuclei and ZAP plug into the same adapter protocol when they are on PATH. I chose not to write a detection engine; the value is the layer above one.

One real run: the row that stopped lying

I could have refused trivy outright on a target with no lockfile. On Juice Shop that would have thrown away its misconfiguration and secret checks, which found 94 findings, to report one gap. So trivy's preflight now declares a partial refusal: the scanner runs, and the one capability it cannot use is named.

textsec modules on OWASP Juice Shop, after the fix
  M2            2/2        0      0         1            0  blind
         blind spot: trivy could not check dependency CVEs on this target: no lockfile or
         other resolvable manifest was found, so trivy has no package versions to match
         against its database — zero dependency findings here means nothing was read, not
         that nothing is vulnerable (package.json alone states ranges, not versions)

No score went up. The same sentence reaches the report:

The ai-sec showcase's "Never a quiet pass" page: trivy with no lockfile is a partial refusal graded blind, and a Could not evaluate block lists trivy, an unsupported language and module M11

Get it, and what it does not do

Try it
git clone https://gitlab.com/agentic-fx/ai-sec.git
cd ai-sec
node bin/sec.mjs doctor
node bin/sec.mjs scan mock --slug demo

It needs Node 20+ and nothing else. mock/ is a read-only app with planted defects. scan with no tier is T0: it reads files and sends nothing.

It is not a pentest and writes no exploits. Business logic (M11) is no-engine on purpose, and nobody outside the project has reviewed it yet.

Try it

Try it
node bin/sec.mjs scan .
node bin/sec.mjs report

Read the last block first. On Juice Shop, M2 is now honestly blind, but that scan also returned 246 findings, and the SQL injection in the login route was not on the first screen. Part 2, The SQL Injection Sat Below 188 Test-Suite Findings, reads that report in the order it was built to be read.

Read this with your AI

textPaste into your AI
Read the post "When Zero Findings Means Nothing Was Read".
1. Summarise it in 5 bullets for an engineer who reports security scan results.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. For one scanner I run today, list what it would report if it could not read my code.

FAQ

Does ai-sec need scanners installed?

No. It needs Node 20 or later. Two scanners are built in, builtin for code and probe for a live target; Semgrep, Trivy, Gitleaks, Nuclei and ZAP are used when they are on PATH, and doctor says what each absence costs.

Can I point ai-sec at any website?

No. Only authorized targets: your own code, employer systems with authorization, staging, CTF and deliberately vulnerable practice apps. A host outside the allowlist or a tier above the ceiling in scope.json is refused with exit code 2.

Is ai-sec a pentest?

No. It writes no exploits, business logic is out of reach by design, and nobody outside the project has reviewed it yet.

Sources

  1. ai-sec on GitLab
  2. ai-sec evidence: ran is not read (M18)
  3. OWASP Juice Shop

Was this useful?

Only you and the owner see your answer and rating. Send private feedback

Enjoyed it? Support fx-blog with a coffee.

Comments

Oldest first

No comments yet. Start the conversation.

You sign in first; what you typed stays here. · 0 / 2,000