An LLM judge is a language model asked to grade another model's output, for example "is this math solution correct?" or "which answer is better?". Fine-tuned judges have a short useful life. On the Shelf Life of Fine-Tuned LLM-Judges finds that a judge trained on answers from weaker generators gets worse when it grades answers from newer, stronger generators. The same judge loses nearly 10 points on unseen DeepScaleR questions. The only remedy that paper tests is collecting labels and retraining offline. Current test-time fixes for judges make the judge think longer per item. J1 appends "Wait" tokens to force reflection, and MCTS-Judge runs a tree search over sub-evaluations. Both add sequential decoding cost to every judgment. Neither uses the most obvious signal a deployed judge has, which is the unlabeled stream of responses it is currently grading.
We propose to train, offline, a small network that reads a batch of the judge's internal activations on the incoming responses. In one forward pass it outputs a low-rank correction that is added to the judge's hidden states. No gradient steps run at deployment, so the extra cost is a fraction of one judge forward pass. The correction is learned across many simulated "deployment episodes", each one a different generator model paired with a different question pool. It is then tested on generators and questions never seen in training. In short, the judge looks at what the new model's answers look like as a group and adjusts itself once, instead of being retrained or reasoning longer about each answer.
Core Claim
The degradation of fine-tuned judges under generator shift has two possible components. One is a calibration shift, where the judge's scores move but it still ranks correct answers above incorrect ones. The other is a ranking shift, where the judge stops telling them apart. A one-number threshold fix can repair calibration shift but not ranking shift. We claim three things. First, a meaningful part of the shelf-life loss is ranking shift. Second, an amortized network that conditions on the target distribution can recover much of that loss without labels. Third, it can do so at well under the latency of either reflection-based scaling or iterative test-time optimization.
The first claim is a check that can fail. If the drop turns out to be almost entirely calibration, the right paper is a short one about prior correction, and we would write that paper instead.
Relation to Prior Work
Three papers motivate the gap. On the Shelf Life of Fine-Tuned LLM-Judges measures negative future-proofing and a "refresh advantage" of up to 7.6 points from retraining on strong-generator data. It never adapts at test time. JETTS shows the downstream cost of this weakness: 8B judges have negative helpfulness when they rerank outputs from 70B generators. LLM-as-a-Judge on a Budget decides how many queries to spend on each item but treats the judge as a fixed oracle. That work is complementary to ours, since a corrected judge could be plugged into its query-allocation scheme.
Two lines of work spend extra test-time compute per instance through token generation. J1 does it with reflection cycles, which its authors observe can flip the judge's criteria. MCTS-Judge does it with tree search, which raises a code judge from 41.0% to 80.0% on APPS but is confined to offline grading. Our method spends no extra tokens and is applied once per batch rather than once per item.
The method itself comes from adaptation work outside language evaluation. Any-Shift Prompting trains an inference network that looks at a test distribution and emits a prompt for a vision-language model without backpropagation. We transfer this amortization idea to a text judge's residual stream, where the residual stream is the running hidden-state vector passed between transformer layers. Amulet shows that closed-form, gradient-free adaptation of an LLM can run about 17 times faster than iterative search. However, it steers generation with an explicit preference prompt, and it fails when the base model has no initial response to that prompt. A judge facing an unknown stronger generator has no such prompt, so we learn the correction from data rather than deriving it.
Batch-statistic methods impose a constraint and also supply our strongest cheap baseline. The work of Mirza et al. and TEMA from Su et al. adapt models purely by updating running statistics during the forward pass. If simple re-centering of activations matches our learned correction, the inference network is not needed.
Research Program
The judge is an 8B open model fine-tuned following the shelf-life recipe. Concretely, we apply supervised fine-tuning and then preference optimization on correctness-labeled answers from weak generators (Llama-3.1-8B, Gemma-2-9B) to DeepScaleR and MMLU-Pro questions. Both datasets have checkable answers, so correctness labels are automatic.
From a pool of open generators of varied strength, together with disjoint question pools, we build a large set of episodes. Each episode contains a generator, a set of questions, that generator's answers, and the correctness labels. We hold out entire generators, including the strong Llama-3.3-70B and Qwen-2.5-32B, and entire question pools for testing.
For a context batch $S$ of target pairs, let $\bar h(S)$ and $\Sigma(S)$ be the mean and a low-rank summary of the covariance of the judge's hidden states at a few chosen layers, read at the verdict position. A small set-encoder $g_\phi$ maps these statistics to coefficients, and each layer's hidden state $h$ is replaced by
$$h' = h + U\, g_\phi\big(\bar h(S), \Sigma(S)\big),$$
where $U$ is a learned low-rank basis shared across episodes. The judge's weights stay frozen. We train $\phi$ and $U$ by minimizing the judge's verdict cross-entropy on each episode's labels. Labels are used only during this offline training. At deployment, $S$ is a sliding window of the unlabeled stream. The activations it needs are already computed when the judge grades those items, so the added cost is the small encoder plus the vector addition.
The analysis has three parts. First, we decompose the shelf-life loss into ranking and calibration components by comparing area under the ROC curve (AUROC), which measures how well the judge separates correct from incorrect answers regardless of threshold, with accuracy at the best threshold. Second, we measure how recovery depends on context size, from 1 to 8 to 64 items. Third, we test robustness on mixed or abruptly switching streams, the error-accumulation risk noted in continual adaptation work such as Hybrid-TTA.
Baselines
The naive reference is the frozen fine-tuned judge with no adaptation, evaluated on held-out strong generators and unseen questions. Everything else is measured relative to it. The ceiling is the shelf-life paper's refresh: fine-tuning the judge on labeled data from the held-out target generator. That paper reports a refresh advantage of up to 7.6 absolute points for Ministral-8B with fine-tuning plus preference optimization. For the exact held-out pairings we use, we will recompute the refresh number ourselves, because the file reports no per-split future-proofing numbers.
The strongest label-free competitor is reflection-based test-time scaling: J1-7B with one to five "Wait" cycles. That paper reports 75.98% overall across its reward benchmarks and 4.8% over RISE-Judge. No reported result was found on Lacuna for J1 on DeepScaleR or MMLU-Pro under generator shift, so we will run it ourselves and report accuracy against wall-clock latency.
The strongest cheap baseline is forward-only statistic correction in the style of TEMA: re-center the same hidden-state layers to the running target mean, with no learned network. Iterative test-time entropy minimization on a low-rank adapter, with backpropagation per batch, is the slow but strong adaptive competitor.
Two ablations isolate the key ingredient. The first, multi-generator fine-tuning, trains the judge directly on the union of all meta-training episodes. This separates the benefit of conditioning on the target batch from the benefit of simply having seen more generators. The second, unconditioned steering, learns a single fixed vector $U c$ with no dependence on $S$. A third ablation restricts the network's output to a bias on the verdict logit, which reduces our method to learned recalibration.
All methods use the same judge, the same episode data where they use any, and an equal hyperparameter budget tuned on held-out validation generators. Beating a baseline means a higher held-out accuracy and AUROC, averaged over seeds, with non-overlapping confidence intervals, at no more than 1.2 times the frozen judge's latency.
Why not just…?
Why not just recalibrate the threshold or correct for label-prior shift? This is the most serious threat. It is the first thing the AUROC decomposition tests. If held-out AUROC barely drops under shift, recalibration recovers nearly everything, and we would report that finding as the result. The bias-only ablation runs this alternative directly. Our method is justified only to the extent that the full correction beats the bias-only version on AUROC, not merely on accuracy.
Why not just put unlabeled or few-shot examples from the new generator in the prompt? Prompting with explicit instructions fails to remove hidden judge heuristics. The Silent Judge finds up to a +30% verdict shift despite instructions, with judges never acknowledging the cue. PersonaEval finds only minor gains from few-shot prompting. Long in-context batches also multiply per-item token cost. We include a few-shot, target-sampled prompt baseline at matched latency.
Why not just let the judge reason longer, through reflection or self-consistency? J1 shows reflection helps but adds sequential latency and can switch the judge's criteria. Parallel voting tends to repeat the base model's initial biases, according to both J1 and PersonaEval. More importantly, per-item reasoning cannot see that a whole stream comes from an unfamiliar generator. Our accuracy-versus-latency curves against J1 settle whether batch-level information beats per-item compute.
Why not just retrain offline, or run test-time gradient steps? Retraining needs target labels and lags behind each new generator release, as the shelf-life paper shows. Iterative test-time optimization multiplies latency, and Amulet reports a roughly 17 times slowdown for iterative methods such as RAIN. Both serve as upper references in our comparison. The question is how much of their gain we recover at a fraction of their cost.
Proposed Experiments and Decisive Checks
The main evaluation uses leave-one-generator-out testing on DeepScaleR and MMLU-Pro. We report held-out accuracy, AUROC, expected calibration error, and per-judgment latency, sweeping context size and mixing streams. The downstream test uses JETTS best-of-N reranking with 70B generators on GSM8k, MATH, and MBPP+, measured by normalized helpfulness. RewardBench serves as a no-harm check on in-distribution preference judgments.
Three checks decide the project. First, the AUROC decomposition must show a ranking component of at least a few points; otherwise we pivot to recalibration. Second, the full method must beat both multi-generator fine-tuning and statistic re-centering on held-out strong generators, recovering at least half of the oracle refresh gap within 1.2 times frozen latency. Third, the corrected 8B judge must move JETTS helpfulness with 70B generators from negative to non-negative. Failing the second check while passing the first would mean the gain comes from training data rather than from adaptation, and we would report that.
References
- On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
- LLM-as-a-Judge on a Budget
- J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
- MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
- Any-Shift Prompting for Generalization Over Distributions
- Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
- The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization
- Unraveling Batch Normalization for Realistic Test-Time Adaptation
- Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?