UNDER THE HOOD
A small guard model (Qwen3-0.6B, LoRA fine-tuned) classifies every message as safe, injection, jailbreak or exfiltration in one forward pass. Every breach is detected deterministically, so the game produces correctly labeled training data.
L3: same model, relaxed threshold. The Watcher runs the same guard as bosses 4 and 5, but it only blocks a message when it is nearly certain (threshold 0.99990224 on its attack score), so more attacks get through.
REQUEST PATH
Every message travels left to right (top to bottom on a phone). Numbers are from the gold test set (v1).
Regex
Bosses 2-5
Blocks obvious attack words before anything else runs.
- attacks passed
- 59.4%
- benign blocked
- 10.3%
Input guard
Bosses 3-5
Qwen3-0.6B + LoRA scores every message in one forward pass (int4, CPU).
- attacks passed
- 0.4%
- benign blocked
- 1.8%
- p95 model time
- 2063 ms
- untuned base passed
- 100.0%
Vault agent
All bosses
A Groq-hosted open LLM holds the boss's secret code and tries to keep it.
- secret
- 8 chars, per player
Output filter
Boss 5
Scans the reply and scrubs it if the code is in there, in any encoding.
- fails
- closed
Leak detector
Every reply
Deterministic check: base64, hex, ROT13, leetspeak, reversed and spaced-out codes all count as a win.
- label
- exact, no model
Scorecard (v1, gold_test)
| Model | ASR | FPR | F1 | AUROC | p95 latency |
|---|---|---|---|---|---|
| regex | 59.4% | 10.3% | 0.531 | 0.652 | n/a |
| llama-prompt-guard-2-86m (Groq) | 51.6% | 10.3% | 0.602 | 0.868 | n/a |
| gemini-judge | 12.8% | 2.8% | 0.915 | 0.969 | n/a |
| deberta-prompt-injection | 10.4% | 14.2% | 0.867 | 0.904 | n/a |
| qwen3-0.6b-zeroshot | 100.0% | 0.4% | 0.000 | 0.591 | n/a |
| qwen3-1.7b-zeroshot | 61.9% | 13.2% | 0.495 | 0.777 | n/a |
| qwen3-0.6b-sft | 0.6% | 1.6% | 0.988 | 0.999 | n/a |
| qwen3-0.6b-sft-dpo | 0.4% | 1.7% | 0.988 | 0.999 | n/a |
| qwen3-1.7b-sft | 0.0% | 1.3% | 0.992 | 1.000 | 7769 ms |
| qwen3-1.7b-sft-dpo | 0.0% | 1.3% | 0.992 | 1.000 | 7769 ms |
| qwen3-0.6b-sft-int4 | 0.6% | 2.1% | 0.985 | 0.999 | 3468 ms |
| qwen3-0.6b-sft-int4-v2 (shipped) | 0.4% | 1.8% | 0.987 | 1.000 | 2063 ms |
| distilbert-student-int8 | 0.3% | 1.8% | - | 1.000 | n/a |
| guard-v2-cascade | 0.4% | 1.1% | - | 1.000 | n/a |
ASR = share of attacks passed (lower is better). FPR = share of benign prompts blocked. In-distribution numbers; see the evaluation report for caveats.
How latency was measured
p95 is the 95th percentile of model time per request (one request at a time, messages capped at 500 characters), shown rounded to the nearest millisecond; n/a means the row was not measured. The shipped guard (qwen3-0.6b-sft-int4-v2) measured 2063 ms on the real host (a Vercel function with one vCPU, warm, one request at a time), so the 300 ms target is not met. A request after a long idle takes about 13 s while the instance boots. Rows measured on other hardware are not comparable: the 1.7B row used a 4-vCPU Kaggle CPU with an fp32 ONNX export at 512 tokens, and it is not the shipped configuration. API models (regex, Prompt Guard, Gemini) have no comparable latency. Numbers come from shared hardware and are noisy.
VERSION TIMELINE
2026-10-30 · SHIPPED
data-v1
19,330 training rows (49.9% attacks, Hindi and Hinglish included), 1,500-row gold test set, leakage-checked splits. (ADR-0030)
2026-10 · SHIPPED
v1 gate: SFT 0.6B is champion
LoRA SFT on Qwen3-0.6B beats every baseline; DPO adds nothing significant, so SFT-only ships. (ADR-0036)
2026-11-01 · SHIPPED
Guard live on a Vercel function
4-bit weight-only ONNX graph on one CPU, its own threshold fitted on val for FPR at most 2%. (ADR-0082)
2026-11-01 · SHIPPED
Leak suite on the live guard
Bosses 3 to 5: 0 leaks in 216 scripted attacks each; boss 1 leaked once in 210. 0% on templated attacks is not unbreakable: human players will find gaps. (ADR-0083)
2026-11-11 · SHIPPED
Cold start tamed
The page wakes the guard on load; a booting guard answers 202 and the game waits for it. (ADR-0085)
2026-11-11 · SHIPPED
Base guard and opener gap measured
The untuned base guard lets 99% of attacks through; short benign openers are the main false-positive gap. (ADR-0090)
2026-11 · NOW
v1.1 Neon Arcade
Bosses, points, hint tokens, skips and the verdict pill with the guard's attack probability. (ADR-0091)
v2 · PLANNED
Guard v2: benign Hindi, Hinglish and short openers
New benign training rows and a retrain to fix the false positives on short and Hindi messages. (ADR-0090)
THE FLYWHEEL
capture -> label (leak / judge / human) -> build dataset -> train (SFT, mining, DPO) ^ | | v shadow 48h <- promote (alias champion) <- eval gate <-------- challenger
OPEN ARTIFACTS
Live metrics snapshot: not published yet.