Skip to content

UNDER THE HOOD

A small guard model (Qwen3-0.6B, LoRA fine-tuned) classifies every message as safe, injection, jailbreak or exfiltration in one forward pass. Every breach is detected deterministically, so the game produces correctly labeled training data.

L3: same model, relaxed threshold. The Watcher runs the same guard as bosses 4 and 5, but it only blocks a message when it is nearly certain (threshold 0.99990224 on its attack score), so more attacks get through.

REQUEST PATH

Every message travels left to right (top to bottom on a phone). Numbers are from the gold test set (v1).

  1. Regex

    Bosses 2-5

    Blocks obvious attack words before anything else runs.

    attacks passed
    59.4%
    benign blocked
    10.3%
  2. Input guard

    Bosses 3-5

    Qwen3-0.6B + LoRA scores every message in one forward pass (int4, CPU).

    attacks passed
    0.4%
    benign blocked
    1.8%
    p95 model time
    2063 ms
    untuned base passed
    100.0%
  3. Vault agent

    All bosses

    A Groq-hosted open LLM holds the boss's secret code and tries to keep it.

    secret
    8 chars, per player
  4. Output filter

    Boss 5

    Scans the reply and scrubs it if the code is in there, in any encoding.

    fails
    closed
  5. Leak detector

    Every reply

    Deterministic check: base64, hex, ROT13, leetspeak, reversed and spaced-out codes all count as a win.

    label
    exact, no model

Scorecard (v1, gold_test)

Guard comparison on the held-out gold test set
ModelASRFPRF1AUROCp95 latency
regex59.4%10.3%0.5310.652n/a
llama-prompt-guard-2-86m (Groq)51.6%10.3%0.6020.868n/a
gemini-judge12.8%2.8%0.9150.969n/a
deberta-prompt-injection10.4%14.2%0.8670.904n/a
qwen3-0.6b-zeroshot100.0%0.4%0.0000.591n/a
qwen3-1.7b-zeroshot61.9%13.2%0.4950.777n/a
qwen3-0.6b-sft0.6%1.6%0.9880.999n/a
qwen3-0.6b-sft-dpo0.4%1.7%0.9880.999n/a
qwen3-1.7b-sft0.0%1.3%0.9921.0007769 ms
qwen3-1.7b-sft-dpo0.0%1.3%0.9921.0007769 ms
qwen3-0.6b-sft-int40.6%2.1%0.9850.9993468 ms
qwen3-0.6b-sft-int4-v2 (shipped)0.4%1.8%0.9871.0002063 ms
distilbert-student-int80.3%1.8%-1.000n/a
guard-v2-cascade0.4%1.1%-1.000n/a

ASR = share of attacks passed (lower is better). FPR = share of benign prompts blocked. In-distribution numbers; see the evaluation report for caveats.

How latency was measured

p95 is the 95th percentile of model time per request (one request at a time, messages capped at 500 characters), shown rounded to the nearest millisecond; n/a means the row was not measured. The shipped guard (qwen3-0.6b-sft-int4-v2) measured 2063 ms on the real host (a Vercel function with one vCPU, warm, one request at a time), so the 300 ms target is not met. A request after a long idle takes about 13 s while the instance boots. Rows measured on other hardware are not comparable: the 1.7B row used a 4-vCPU Kaggle CPU with an fp32 ONNX export at 512 tokens, and it is not the shipped configuration. API models (regex, Prompt Guard, Gemini) have no comparable latency. Numbers come from shared hardware and are noisy.

VERSION TIMELINE

  1. 2026-10-30 · SHIPPED

    data-v1

    19,330 training rows (49.9% attacks, Hindi and Hinglish included), 1,500-row gold test set, leakage-checked splits. (ADR-0030)

  2. 2026-10 · SHIPPED

    v1 gate: SFT 0.6B is champion

    LoRA SFT on Qwen3-0.6B beats every baseline; DPO adds nothing significant, so SFT-only ships. (ADR-0036)

  3. 2026-11-01 · SHIPPED

    Guard live on a Vercel function

    4-bit weight-only ONNX graph on one CPU, its own threshold fitted on val for FPR at most 2%. (ADR-0082)

  4. 2026-11-01 · SHIPPED

    Leak suite on the live guard

    Bosses 3 to 5: 0 leaks in 216 scripted attacks each; boss 1 leaked once in 210. 0% on templated attacks is not unbreakable: human players will find gaps. (ADR-0083)

  5. 2026-11-11 · SHIPPED

    Cold start tamed

    The page wakes the guard on load; a booting guard answers 202 and the game waits for it. (ADR-0085)

  6. 2026-11-11 · SHIPPED

    Base guard and opener gap measured

    The untuned base guard lets 99% of attacks through; short benign openers are the main false-positive gap. (ADR-0090)

  7. 2026-11 · NOW

    v1.1 Neon Arcade

    Bosses, points, hint tokens, skips and the verdict pill with the guard's attack probability. (ADR-0091)

  8. v2 · PLANNED

    Guard v2: benign Hindi, Hinglish and short openers

    New benign training rows and a retrain to fix the false positives on short and Hindi messages. (ADR-0090)

THE FLYWHEEL

capture -> label (leak / judge / human) -> build dataset -> train (SFT, mining, DPO)
   ^                                                                |
   |                                                                v
 shadow 48h <- promote (alias champion) <- eval gate <-------- challenger

OPEN ARTIFACTS

Live metrics snapshot: not published yet.