---
title: "Grade Your Security Scanner Against Answers It Didn't Write"
author: "Harry"
published: 2026-11-02T01:59:25.639Z
updated: 2026-11-02T10:46:04.971Z
category: "Security Testing"
tags: ["ai-sec"]
canonical: "https://fxblog.dev/posts/security-scanner-benchmark-answer-key"
---

# Grade Your Security Scanner Against Answers It Didn't Write

## Key takeaways

- A benchmark you wrote can only award self-measured; ai-sec grades a module measured only on a corpus nobody on the project wrote.
- OWASP Juice Shop annotates each planted defect in its own source, and benchmarks/juiceshop.mjs re-derives the answer key from a clone.
- The first ai-sec run against Juice Shop found 2 of 27 annotated lines, and eight rules written from the misses raised it to 16 of 27.
- The benchmarks exposed a report that was 76% test suite, a rule naming the wrong cause, a merge that discarded findings and a scorer that never read occurrences.
- Defects a scanner cannot find stay in the answer key with a written reason, and the benchmark test fails if more than half the key is out of reach.

> **TL;DR** A fixture you wrote can only agree with you, so a scanner that scores full marks on it has proved nothing. A security scanner benchmark is worth something when someone else wrote the target, and better still, the answers. Only here: ai-sec re-derives its OWASP Juice Shop answer key from Juice Shop's own source annotations, and the first run against it found 2 of 27.
>
> **Do this:** Clone OWASP NodeGoat, copy in ai-sec's committed answer key, scan it read-only at T0, and run `sec modules`.

[Part 2](02-security-scan-report.md), **The SQL Injection Sat Below 188 Test-Suite Findings**, ended on nine modules graded `measured` on Juice Shop: full marks on a corpus nobody here wrote. This part opens the answer keys behind that grade, and the first run before any rule was added.

`mock/`, the app I wrote with planted defects, gives every module full marks. OWASP NodeGoat gave three modules zero. Nothing in the ruleset changed between those two runs. Only the target did.

**This series, in three parts:**

1. **When Zero Findings Means Nothing Was Read**
2. **The SQL Injection Sat Below 188 Test-Suite Findings**
3. **Grade Your Security Scanner Against Answers It Didn't Write** (you are here)

## Four benchmarks, because they answer different questions

|                       | target written by | answers written by     | evidence         | best grade      |
| --------------------- | ----------------- | ---------------------- | ---------------- | --------------- |
| `mock/`               | us                | us                     | reading files    | `self-measured` |
| OWASP NodeGoat        | someone else      | us, from its tutorials | reading files    | `measured`      |
| OWASP Juice Shop      | someone else      | the target itself      | reading files    | `measured`      |
| `benchmarks/dynamic/` | us                | us                     | sending requests | `self-measured` |

## Juice Shop writes its own answers

Juice Shop marks the exact line of each planted defect in its own source:

```ts
models.sequelize.query(`SELECT ... '${req.body.email}' ...`) // vuln-code-snippet vuln-line loginAdminChallenge
```

`benchmarks/juiceshop.mjs` reads the 27 markers back out of a clone, so the key is re-derived, not trusted. On NodeGoat I chose which defects to record. Here I chose nothing.

The first run found **2 of 27**, both SQL injection, both a shape `mock/` already had. The misses were the lesson: Angular's `bypassSecurityTrustHtml`, an allowlist checked with `url.includes`, a guard commented out one character from live, a coupon tool whose limit lived only in a sentence to the model. Eight rules later, each written to generalise past this corpus: **16 of 27**.

![The showcase grades page: seven grades from measured to no-engine, and sec modules on Juice Shop showing M2 blind, M4 and M7 measured, M11 no-engine, 3 partial, 1 blind, 9 measured, 1 no-engine](https://fxblog.dev/media/01a199f8-2d9e-7474-aead-2df25bf25ccc/1600)

_The answers for Juice Shop are its own_

## What the answer keys broke

| What broke                            | What showed it                                                                                                                  |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| The report was 76% test suite         | 188 of 246 findings in `test/` (part 2)                                                                                         |
| A rule named the wrong cause          | `nosql-operator-injection` fired on a Sequelize `where:`, which binds the value. The real defect was whose wallet gets credited |
| The merge discarded findings          | the dynamic benchmark reported 4 of 7 header defects; `probe` had found all 7, and a shared CWE family folded three away        |
| `occurrences` was never scored        | a finding anchored at the cause, listing the right call site, counted as a miss                                                 |
| Any `expected.json` was an answer key | a repo with its own golden file would have been graded against somebody else's data                                             |

## Questions it cannot answer stay in

Three Juice Shop defects are still unfound, and I kept them in the denominator:

- unguarded frontend routes: a route without a guard is the normal case;
- crypto addresses in the redirect allowlist: a business question;
- a review update that ignores the owner: the variable is used, just not for authorization.

Deleting the questions you cannot answer turns a benchmark into a decoration, so the benchmark test fails if more than half the key is out of reach. Two answers that moved to "found" after the rules were written carry a `was_missed_until` field saying why.

## A target that is meant to be attacked

The dynamic benchmark starts a vulnerable server I wrote on a loopback port, scans it at T2 and stops it. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its own `scope.json`.

Cross-account read: three requests, in this order

```mermaid
flowchart LR
  A[as the owner<br/>what the object looks like] --> B[with no session<br/>the control]
  B --> C[as the other account<br/>the test]
  B -- object still returned --> P[public: a different defect]
  C -- same body digest --> I[cross-account read]
```

Two endpoints are correct on purpose. `/api/inbox` answers 200 to every account with that account's own data, so a check that reads only the status code reports it falsely.

> **Warning**
>
> Juice Shop is a teaching app, the dynamic tiers have faced only a target I wrote, and nobody outside the project has reviewed ai-sec. Those are the limits of every grade above.

## Try it

NodeGoat is a deliberately vulnerable practice app. It is scanned read-only at T0 and never run.

```sh
git clone --depth 1 https://github.com/OWASP/NodeGoat.git /tmp/nodegoat
cp benchmarks/nodegoat-expected.json /tmp/nodegoat/expected.json
node bin/sec.mjs scan /tmp/nodegoat --slug ng
node bin/sec.mjs modules --slug ng
```

Without `trivy` on PATH, M2 and M10 grade `failing` there, and the ledger says why.

## Read this with your AI

```text
Read the post "Grade Your Security Scanner Against Answers It Didn't Write".
1. Summarise it in 5 bullets for an engineer who owns a security scanning setup.
2. Ask me 3 questions to check I understood; wait, then correct me.
3. Name one public practice app I could use as an answer key for a scanner I run.
```

> **Success**
>
> You can tell a grade earned on your own fixture from one earned on answers someone else wrote. Start with `node bin/sec.mjs doctor` on a repo you own.

## FAQ

### Why scan OWASP Juice Shop and NodeGoat?

They are deliberately vulnerable practice applications that someone else wrote, so they can surprise the scanner. ai-sec scans a clone read-only at T0 and never runs them.

### Does the dynamic benchmark attack anything?

Only a server ai-sec starts itself from its own repository. The app refuses to bind anywhere but 127.0.0.1, and the harness checks the target is that server before writing its scope.
