Stealing the Judge's Rubric: Inverting Hidden Evaluation Prompts from Unfaithful Critiques

When a language model grades an essay, a research paper, or a customer-service reply, the grading criteria usually live in a hidden system prompt — a rubric the evaluated party never sees. Two recent findings make those secrets both valuable and exposed. First, they are valuable: when a retrieval system was secretly given the evaluator's prompt template and gold facts, its measured quality jumped by 16% to 100% across metrics on the TREC NeuCLIR 2024 collection Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?. Second, they are exposed: a text-only encoder trained on ordinary model completions can reconstruct the system prompt that produced them with 96.7 cosine similarity, without any access to token probabilities Extracting Prompts by Inverting LLM Outputs. What nobody has done is put these together: have the evaluated side passively collect the judge's critiques and verdicts over normal, benign interactions, and train an inversion model to reconstruct the hidden rubric from them — then cash the recovered rubric in for higher scores.

Hallucination enters as the mechanism that decides whether this attack works. Judges demonstrably fabricate content-grounded justifications for verdicts that were actually driven by hidden prompt conditions: when verdicts were swung by provenance or recency labels, the chain-of-thought acknowledged the cue 0% of the time The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge, and when system-prompt "stakes" framing shifted verdicts by up to −9.8 percentage points, the reasoning never once mentioned it ($ERR_J = 0.000$) Context Over Content: Exposing Evaluation Faking in Automated Judges. Judges also invent nonexistent flaws in valid texts Can We Automate Scientific Reviewing?. The open two-sided question is whether this fabricated, unfaithful commentary masks the hidden rubric from inversion (noise) or leaks it (rubric-conditioned boilerplate that inversion models can exploit). No reported result found on Lacuna for inversion of a judge's rubric from its critiques; the closest works stop short on all three sides of this question, as laid out below.

Core Claim

The claim is twofold. First, black-box prompt inversion transfers from ordinary assistant outputs to judge transcripts: an encoder that ingests (query, candidate response, critique, verdict) tuples can reconstruct the hidden evaluation prompt well enough to measurably inflate the evaluated system's scores — closing part of the gap between the rubric-blind baseline and the oracle-secret ceiling that Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? established by hand-feeding the secrets. Second, and more surprising, the judge's hallucinated rationales are a leakage channel rather than a cover: because unfaithful justifications are generated by the same rubric-conditioned model, they carry rubric vocabulary and phrasing that faithful, purely content-driven critiques may not. If this second claim fails — if hallucinated critiques are harder to invert than faithful ones — that is itself a publishable negative result with direct defensive value (judges could deliberately rationalize in rubric-neutral language), and the experimental design below separates the two outcomes cleanly.

This sits precisely in the unexplored cell of the existing literature. Optimization-based Prompt Injection Attack to LLM-as-a-Judge forces a verdict for one adversarial answer via gradient-optimized suffixes (attack success rate above 90% on Mistral-7B and Llama-3-8B judges) but never extracts or reconstructs the judge's instructions. Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? measures the payoff of knowing evaluation secrets but supplies them artificially rather than stealing them. Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation shows that rubric text alone predicts verdicts (a rubric-only classifier reaches balanced accuracy above 0.8, blinded to the response) and that judges are inert under counterfactual reversals — evidence that rubrics strongly shape verdicts, but it never inverts or reconstructs them. The inversion literature itself — Better Language Model Inversion by Compactly Representing Next-Token Distributions and Extracting Prompts by Inverting LLM Outputs — targets assistant system prompts from user-facing completions, never evaluation prompts from critique transcripts.

Research Program

The program builds one pipeline, which we call rubric inversion, in four parts. First, a transcript corpus: we take real evaluation prompts (rubrics from the HEALTHBENCH family and RESEARCHRUBRICS, plus custom-instruction prompts of the kind used in Extracting Prompts by Inverting LLM Outputs) and pair each with candidate responses and queries drawn from MT-Bench, ELI5, and HEALTHBENCH items, plus multi-evidence factuality items from the ME-FEVER set used by Halu-J: Critique-Based Hallucination Judge. Open-weight judges (Gemma and LLaMA families, following the judge suite in Judging LLM-as-a-Judge) produce verdict-plus-critique transcripts; using open judges also lets us run logit-based inversion as an upper bound where token probabilities are exposed. Second, the inversion model: a T5-base sparse encoder of the output2prompt type, trained on roughly 30,000 transcript–rubric pairs — the sample budget that Extracting Prompts by Inverting LLM Outputs found sufficient, two orders of magnitude below older logit methods. Third, the exploit loop: the recovered rubric is fed to the evaluated generator as rewriting guidance, and we measure score inflation versus a rubric-blind generator, mirroring the CRUCIBLE probes but with a stolen rather than granted secret. Fourth, the hallucination mechanism study: we manipulate rationale faithfulness directly — injecting provenance and recency shortcut cues as in The Silent Judge and stakes framing as in Context Over Content — and compare inversion accuracy on cue-driven (hallucinated-rationale) transcripts versus content-driven ones, with cue acknowledgment verified to remain near zero so that the split is clean.

The headline quantity is the gaming gain, defined as the judge-assigned score of a generator rewriting with the recovered rubric minus the score of the same generator, same compute, rewriting blind:

$$\Delta = S(\text{candidate} \mid \hat{R}) - S(\text{candidate} \mid \varnothing),$$

where $\hat{R}$ is the reconstructed rubric and $S$ the judge's score on the same items. The oracle ceiling from Insider Knowledge — where known evaluation prompts lifted metrics by +16% to +100% and near-oracle filtering pushed combined scores to 0.97–0.99 — tells us what fraction of the ceiling a stolen rubric captures.

Baselines

The do-nothing reference point is the rubric-blind generator: the standard evaluation setting where the evaluated system knows nothing about the judge's criteria. Any positive $\Delta$ from a recovered rubric beats it trivially, but it anchors the scale; the oracle-secret condition from Insider Knowledge anchors the top. The strongest existing competitor for the inversion itself is the zero-shot transfer of the pretrained output2prompt encoder, which Extracting Prompts by Inverting LLM Outputs reports inverting unseen models (GPT-4, Mistral-7B, Gemma-2B) at cosine similarity above 92, and 96.7 CS on its own target distribution. We will run it unchanged on our judge transcripts — same data, no extra tuning — and also fine-tune it on judge-domain transcripts with the same training budget our new encoder gets, so the comparison isolates domain shift rather than model capacity; beating it means higher CS and RougeL on held-out rubrics and converting that into significantly larger $\Delta$ where it does not. A second serious competitor is prompted rubric inference: a strong general-purpose LLM given the same transcripts and asked to write out the judge's criteria, with an equal prompt-iteration budget; no reported result found on Lacuna for this approach, which is exactly why it must be run rather than dismissed. The key ablation isolates the proposal's central ingredient: inverting from verdicts alone versus verdict-plus-critique, and, within critiques, faithful versus hallucinated-rationale subsets. If verdict-only inversion matches verdict-plus-critique, the hallucination story is moot and the proposal reduces to a domain-transfer study; we report whichever way it lands.

Why not just…?

Why not just ask the judge to reveal its instructions? Direct extraction fails against defended judges in exactly the setting that matters: Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications shows defended GPT-4-1106 dropped worst-case extraction success from 99% to 3%, and while open models stayed largely indefensible, a production judge can be a defended frontier model. Worse, asking is detectable; benign transcripts collected during ordinary evaluation are not. Our pipeline needs no cooperation from the judge at all.

Why not just read the judge's chain-of-thought? Because the chain-of-thought is the documented site of the cover-up: cue acknowledgment was 0.00% across all experiments in The Silent Judge and evaluation recognition was 0.000 across all reasoning-model judgments in Context Over Content. Human reading of critiques recovers the rationalization, not the rubric. That said, a strong reader (or prompted LLM) might still infer partial criteria — that is the real threat, which is why prompted rubric inference is a first-class baseline above rather than an argument we wave away; the trained-encoder-versus-prompted-LLM comparison on identical transcripts settles it empirically.

Why not just run JudgeDeceiver-style injection instead? Optimization-based Prompt Injection Attack to LLM-as-a-Judge is the honest threat here: its attack success rate exceeds 90% on Mistral-7B and Llama-3-8B judges, beating AutoDAN by over 37% on MT-Bench. But it solves a different problem — forcing one verdict for one optimized answer — while requiring gradient access, shadow models, and per-instance optimization, and it teaches you nothing reusable about the criteria. Rubric inversion is passive, works on text-only APIs, and yields a persistent asset that helps every future submission, including ones you cannot adversarially edit. We will nevertheless report both attack paths side by side on MT-Bench (cost, detectability, score gain) so a reader can judge the trade-off directly rather than take our word for it.

Why not just use logit-based inversion, which is already stronger? Where token probabilities are exposed, Better Language Model Inversion by Compactly Representing Next-Token Distributions reaches 51% exact prompt recovery on Llama-2 Chat and 60.5% exact match out-of-distribution — but production judge APIs increasingly suppress logprobs, the constraint that motivated the text-only approach of Extracting Prompts by Inverting LLM Outputs. We use PILS on our open-weight judges purely as an upper bound and expect the text-only path to be the practically relevant one; if logit access turns out to be commonly available on real judge deployments, that shifts the framing, not the conclusion.

Decisive Checks

Three results decide the proposal. First, on held-out rubrics, domain-trained inversion must beat zero-shot output2prompt transfer (CS above 92 reported for cross-model transfer in Extracting Prompts by Inverting LLM Outputs) by a clear margin and approach the oracle-secret payoff structure of Insider Knowledge. Second, the faithfulness split must show a significant difference in inversion accuracy between hallucinated-rationale and content-driven transcripts, in whichever direction — this is the falsifiable core of the hallucination claim, supported on the leakage side by DORY's finding that output uncertainty drops 40–60.7% on tokens shared with the hidden prompt DORY: Deliberative Prompt Recovery for LLM. Third, a defense probe: paraphrasing the judge's rubric or deploying ProxyPrompt-style continuous prompts ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks, which reports a 94.70% protection rate against direct extraction, must be tested against our inversion, since benign-transcript inversion may sidestep defenses built for adversarial asking — and rubric paraphrase in particular interacts with the inertia findings of Judging LLM-as-a-Judge in a way nobody has measured.