Measuring prompt-injection detection honestly

The number we started with

For a long time, Koreshield had no measured precision or recall. Detection changes shipped on the strength of individual examples, and the only corpus we had, 302 internal examples, was also the training data for the semantic layer. Every figure computed on it above 60% depended on that leakage. Testing a detector on its training data does not measure detection. It measures memory.

When we measured the rules engine on a public dataset it had never seen, deepset/prompt-injections, it caught 2 of 263 attacks: 0.8% recall. It also raised zero false positives on the 399 benign examples. Very precise, almost blind. Even counting any indicator at all as a block, the ceiling of that engine on the external set was 19% recall.

We publish that number because every later improvement is only meaningful against it.

An evaluation that cannot flatter us

Every detection change is now judged by the same evaluation harness, using the same decision functions the scan endpoints use. Its rules are deliberately hard to game:

• Near-duplicate internal examples are grouped first, so paraphrases of one attack cannot land on both sides of a split.
• Groups are assigned deterministically to train, calibration and test (60 / 20 / 20), and only the test split is reported.
• The test split is frozen by a hash of its example IDs, so it cannot change quietly.
• External datasets are pinned and verified by checksum, and used for testing only.
• Recall, precision, false-positive rate and F1 are reported per dataset and boundary, each with a 95% confidence interval, alongside latency percentiles.
• A gate fails any change that drops recall by more than one point or raises the false-positive rate by more than 0.3 points against the recorded baseline.

What a calibrated classifier changed

The measurements were clear that the recall lever is a model, not more rules. Koreshield now scores requests with rules plus a pinned open-source classifier (protectai/deberta-v3-base-prompt-injection-v2, Apache-2.0), with its threshold chosen on calibration data only, at a 1% false-positive target. No reported number is tuned on its own test rows.

On held-out test rows, the deployed variant changed the picture for direct attacks. On Lakera Gandalf, rules alone caught 2.8% of attacks and rules plus classifier caught 96.2%. On SPML, the figures were 4.5% and 95.4%. On our internal test split, recall rose from 9.5% to 35.7%.

What it costs on support traffic

Koreshield's customers run support agents, so a false block is a customer who cannot get help. That is why the evaluation includes ordinary customer-support messages, not only attack datasets. On 994 held-out support messages, rules alone flagged none; the deployed classifier variant flagged 10, or 1.0%.

Holding false positives near 1% on support traffic has a price, and the next section is about it.

What it still misses

We would rather you hear the limits from us:

• Subtle and indirect injections. On the deepset set, the deployed variant catches 10% of attacks at a support-safe threshold. Direct "ignore your instructions" attacks are caught at 95 to 99%; these are not.
• Languages other than English. The model is English-only, and on the German deepset rows it caught 1 of 23 attacks.
• Long inputs. They dominate latency, so context scans cap classifier time at 200 ms per request and mark the remaining documents as degraded.

Shadow before enforce

The classifier ships in shadow mode by default. In shadow, its verdict is stored in the evidence but never changes a decision, which gives each workspace a side-by-side record of what the current engine decided and what the classifier would have decided. Switching the engine needs a management-scoped key and is audited. Our guidance is at least 14 days of shadow traffic per workspace, with every disagreement reviewed, before the classifier enforces.

Two further design choices keep the system honest. Every event records an engine version, combining the rule-set fingerprint with the model hash prefix and threshold, so any decision can be traced to the exact engine that made it. And classifier probabilities are never returned to callers, because a score would give an attacker an oracle for tuning evasions.

Ask for the same from anyone

If you are evaluating AI security for a support workflow, ask any vendor, us included, for recall on data the detector was not built from, the false-positive rate on traffic like yours, and what happens when the detector cannot decide in time. Then run it in detect mode against your own traffic and judge it on evidence.