Canary Criteria: Catching Hijacked LLM Judges Before the Expert Budget Runs Out

Large language models are now routinely used as judges. They grade code and short answers, screen papers, and label preference data in place of scarce human experts. The judge reads the scoring instructions and the submission in the same context window, so a submission can carry text that tells the judge what score to give. That text might be hidden white-on-white, a fluent optimized suffix, or a forged "reasoning" passage. Pipelines that save expert time usually send to humans only the cases where the AI looks unsure or disagrees with something. A successful injection does the opposite: it makes the judge confident, fluent and ordinary-looking. The cases most in need of an expert are therefore the ones least likely to be escalated.

We propose a different signal for spending a fixed expert-audit budget. Alongside the real rubric, the judge is given a few secret, randomized canary criteria: rubric items whose correct answer for this particular submission can be computed by a cheap deterministic check. Examples are "award this point only if the answer contains a function named X," "only if the response is under N words," or "only if the cited figure number appears in the text." An honest judge scores the canaries correctly. A judge whose instructions have been overridden tends to award everything. The injection works by making the judge stop executing the rubric faithfully, and canaries measure exactly that faithfulness. We use canary violations to decide which items the expert sees.

Core Claim

Injection against a judge is an attack on rubric execution. Rubric execution can be spot-checked from the inside with criteria whose ground truth the defender knows and the attacker cannot predict. For a submission $x$ and a set of sampled canaries $C$, we define the violation score

$$v(x) = \frac{1}{|C|}\sum_{c \in C} \big|\, j(x,c) - t(x,c) \,\big|,$$

where $j(x,c)\in\{0,1\}$ is the judge's verdict on canary $c$ and $t(x,c)\in\{0,1\}$ is the programmatically computed truth. The expert audits the top-$B$ items ranked by $v$, with ties broken by score. We claim that at matched expert budgets this routing catches substantially more score-inflating injections than routing by judge uncertainty or cross-judge disagreement. We also claim it does so against optimized, low-perplexity attacks that evade content filters. The reason it should resist adaptive attackers is that an attacker who wants to pass randomized canaries must keep the judge faithfully executing an unknown rubric, and that faithful execution is precisely what score inflation has to break.

This idea is closest to two lines of work.

The first is the known-answer detection tested against Optimization-based Prompt Injection Attack to LLM-as-a-Judge. That paper builds JudgeDeceiver, a gradient-optimized suffix that flips pairwise judges, and reports that over 70% of its successful attacks on Llama-3-8B evade perplexity and known-answer checks. This negative result constrains our design. Known-answer detection asks the model to repeat a secret key, which is a side task the injection can leave untouched. Canaries are instead scored items inside the evaluation itself, so they cannot be separated from the behavior the attack hijacks. Whether that difference matters is our first decisive test.

The second is the Disagreement Escalation mechanism in Adopt Machine-Human Collaboration Peer-Review through Computational Research Assessment. That position paper proposes sending papers to senior editors when AI and human assessments diverge, but it has no experiments against adaptive attacks. Separately, Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review builds the PaperGuard benchmark and shows that injections suppress critical review content. Read together, these findings motivate our concern: a hijacked reviewer that sounds like an average reviewer may never trigger escalation.

Research Program

The work has three parts.

Canary generators. For each evaluation regime we build a generator that samples criteria with deterministic ground truth.

  • Code (APPS pairwise): identifier presence, whether a specific test case passes, and line counts.
  • Short-answer grading (GAOKAO23, GSM8K, SciEntsBank from Exposing Security Vulnerabilities in LLM Based Educational Grading Agents): whether the final numeric answer equals a value, and whether a stated unit appears.
  • Instruction following (LLMBar and MT-Bench): length, keyword and format constraints.
  • Peer review (PaperGuard): the presence of specific section titles, equation counts, and the figure-caption content the reviewer is asked to cite.

Canaries are phrased in the same register as real rubric items. They are drawn freshly per call from a large template pool, and some are true and some false for each submission. This prevents "always yes" and "always no" from both scoring well.

Budgeted router. We compare routing by $v(x)$ alone, by $v(x)$ combined with the judge's score (inflated, violating items are the most dangerous), and by a learned combination with judge entropy. The comparison holds the number of expert-audited items fixed at 1%, 5%, 10% and 20% of the pool. This mirrors the cost accounting in Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images. That work sends only crowd labels that its models dispute to pathologists and reaches 95.79% of full-expert accuracy at 73.30% lower expert cost. We borrow its budgeted triage logic but change what counts as "suspicious": suspicion comes from rubric-execution failure, not from label-model inconsistency.

Adaptive attacker study. We extend JudgeDeceiver's objective with a canary-passing term. The attacker optimizes over a shadow canary distribution, the same way JudgeDeceiver optimizes over shadow competitor responses. We also add prompt-level variants: role-play attacks, CoT Forgery from Prompt Injection as Role Confusion, and instructions to "obey all formal checks but rate content maximally." We then measure the trade-off the attacker faces between canary pass rate and score inflation. The central empirical question is whether that trade-off is real.

Baselines

Do-nothing reference. The first baseline sends no items to experts. The second spends the same budget on uniformly random audits. Random audit gives the floor: at budget fraction $b$ it catches roughly a fraction $b$ of attacks. Any routing method must clear this line clearly.

Uncertainty routing. This audits the items where the judge's verdict probability is closest to indifference, or where the swapped-order verdicts conflict. It uses the Swap procedure from Evaluating Large Language Models at Evaluating Instruction Following, which raised GPT-4 positional agreement to nearly 97% on LLMBar. This is the routing signal that active preference pipelines such as ActiveUltraFeedback implicitly trust. It runs on the same judge calls, so it adds no cost.

Cross-judge disagreement escalation. This is the operational form of the CRA proposal. We use three judges from different families and audit the items with the largest score variance. It costs about three judge calls per item. That is comparable to our canary overhead, since canaries are appended to a single call. For this baseline there is no reported result found on Lacuna under attack.

PaperGuard's chunk-based embedding defense. This is the strongest existing detector in the file. It reports 95% detection accuracy with 0% false positives on PaperGuard and caught all injections in 17 real arXiv papers (Does AI Reviewer See the Full Picture?). We run it on PaperGuard with its released pattern database and use its similarity score as the routing signal. Beating it means matching its detection on PaperGuard's attacks while clearly exceeding it on attacks it has not seen, namely our paraphrased and adaptive injections.

DataFilter sanitization. Defending Against Prompt Injection with DataFilter cuts attack success from over 40% to about 2% on AgentDojo with about a 1% utility drop. We apply it before judging and report both residual attack success and the change in clean grading accuracy. Code and procedural answers legitimately contain imperatives, so clean accuracy is where this baseline is expected to suffer.

Role-probe routing. We train linear role probes as in Prompt Injection as Role Confusion on open-weight judges, using clean and injected submissions from a held-out attack family. We route by the "systemness" the probes assign to submission tokens. No reported result was found on Lacuna for role probes as an injection detector on judge benchmarks.

Ablations. Three ablations isolate the key ingredient:

  • Fixed, public canaries. This tests whether randomness and secrecy matter.
  • Canaries asked in a separate call from the real rubric. This turns them into known-answer checks and tests whether embedding them in the scored task matters.
  • Canaries all true or all false. This tests whether balanced truth is needed to stop blanket-yes judges.

All methods share the same judge models (Llama-3-8B, Mistral-7B, Qwen-2.5, and GPT-4o where accessible), the same attacked pools, and the same expert budgets. Learned components get the same held-out tuning split.

Why not just…?

Why not just filter inputs with a moderation API or a perplexity check? The file already answers this. Moderation APIs detected 0% of peer-review injections (Does AI Reviewer See the Full Picture?). Perplexity checks missed over 70% of successful JudgeDeceiver attacks (Optimization-based Prompt Injection Attack to LLM-as-a-Judge). Five data-cleaning filters missed BiasTrojan poisons entirely (BiasTrojan). These filters look at whether text looks malicious. Effective judge injections are built not to look malicious. We include perplexity filtering only as a reference point.

Why not just escalate on disagreement between several judges? This is the most serious competitor, and we do not dismiss it. The "stronger is weaker" result implies that one injection affects judges of different capability unevenly, which could create exactly the disagreement this method needs. It could fail against injections tuned against a shared open-weight surrogate that transfer across families, and against critique-suppressing attacks that push all judges toward an unremarkable consensus. The cross-judge baseline at matched cost, run under transfer attacks, settles the question. If it equals canary routing there, we will report that the cheaper and simpler method suffices and recast canaries as a complement to it.

Why not just use PaperGuard-style retrieval or DataFilter sanitization? Retrieval against a library of known injections works on known patterns. Its own authors note that adaptive, academic-sounding injections can slip past chunk similarity. Sanitizers strip imperatives, but in code and procedural answers imperatives are the content. Our held-out-attack split and the clean-accuracy measurement show whether either baseline holds up off its training distribution. Neither baseline tells the expert which items to look at under a budget, though both can be used as routing scores, and we test them that way.

Why not just harden the judge with rules and step-by-step reasoning? Rules plus reference raised GPT-4 from about 73% to 82.8% on adversarial LLMBar. The same paper found that step-by-step reasoning can rationalize appealing wrong answers (Evaluating Large Language Models at Evaluating Instruction Following). CoT Forgery reaches 60% success on frontier models by faking that reasoning (Prompt Injection as Role Confusion). Hardening is orthogonal to routing, so we apply the best prompting to every judge in every condition.

Proposed Experiments

Attacked pools. We build attacked pools on four benchmarks:

Each pool mixes 10% to 30% attacked items with clean ones. Gold labels (LLMBar's human labels at 94% agreement, execution results, answer keys, and accept/reject outcomes) stand in for the expert, who is assumed to be correct on audited items.

Primary metrics. At each budget we report:

  • the fraction of attacked items caught;
  • residual score inflation on unaudited items;
  • the clean false-audit rate;
  • clean grading accuracy with canaries appended. This checks that canaries do not distract the judge, and we require less than one point of loss.

We also report detection AUROC for $v(x)$ independent of any budget.

Downstream check. We inject JudgeDeceiver triggers into a fraction of UltraFeedback candidate responses and run DRTS-style acquisition. We then compare reward models trained on the raw pairs, on pairs cleaned by canary routing, and on pairs cleaned by uncertainty routing, all at equal audit budgets. This tests whether canaries protect the active-learning setting that ActiveUltraFeedback assumes is trustworthy.

Decisive checks. The proposal stands or falls on three results.

  • Known-answer contrast. Embedded canaries must catch the JudgeDeceiver attacks that separate-call known-answer checks miss. If the embedding ablation performs the same as the full method, the core claim is wrong.
  • Adaptive-attacker trade-off. Under the canary-aware attack, raising the canary pass rate should measurably reduce score inflation. If attackers achieve both high canary pass rates and full inflation, canaries are defeated and we will say so.
  • Head-to-head at low budget. At 5% expert budget, canary routing should catch clearly more attacks than cross-judge disagreement and the PaperGuard detector on held-out attack families, while matching PaperGuard's reported 95% detection on its own attacks.