On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization

Janvijay Singh, Austin Xu, Yilun Zhou, Yefan Zhou, Dilek Hakkani-Tur, Shafiq Joty

TL;DR

Fine-tuning smaller LLMs to act as judges is a cost-effective way to evaluate model outputs, but these judges have a limited "shelf life." This paper reveals that while fine-tuned judges are highly backward-compatible with older models, they fail to generalize to stronger, future generators or entirely new questions. Keeping judges accurate requires continuous data refreshing and continual learning.

What Problem Does This Paper Address?

The "LLM-as-a-judge" paradigm is everywhere—used for automated evaluation, leaderboard scoring, and providing reward signals for RLHF. To avoid the high costs and systematic biases (like length or position bias) of frontier models, practitioners frequently fine-tune smaller, specialized judge models.

However, standard evaluation setups treat this as a static problem. In reality, the AI ecosystem is dynamic: generator models improve rapidly, and evaluation prompts evolve. This paper investigates the shelf life of fine-tuned LLM judges across three critical dimensions:

  1. Future-proofing: Can a judge trained on today's model responses accurately evaluate superior responses from tomorrow's models?
  2. Backward-compatibility: Can a refreshed judge still reliably evaluate older, weaker models?
  3. Question generalization: How well does a judge perform on entirely unseen questions at test time?
Figure 1: High-level overview of our setup for studying Future-Proofing, Backward-Compatibility, and Question Generalization through the lens of generalization and robustness to input distribution shifts. Q and R represent questions and responses, respectively, with responses generated by the shown generator models (Gemma2, Qwen2, Gemma3, Qwen2.5). Future-Proofing evaluates how well judges trained on responses from weaker, older generators (green: Gemma2, Qwen2) assess responses from stronger, newer generators (yellow: Gemma3, Qwen2.5). Backward-Compatibility examines the reverse direction. Question Generalization measures performance on in-distribution questions and corresponding responses that were both not included (dashed Q and R) in the training.

Methods

The authors formalize automatic evaluation as a dual-distribution problem, separating the question distribution (what is being asked) from the response distribution (how models answer).

They evaluate pairwise judges (models tasked with choosing the better of two responses) across:

  • Three backbone architectures: Llama-3.1-8B, Ministral-8B, and Mistral-24B.
  • Three training recipes: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and a combined SFT+DPO loss.
  • Generator tiers: Generators are clustered into "Weak" (e.g., Gemma-2-9B, Qwen-2-7B) and "Strong" (e.g., Gemma-3-12B, Llama-3.3-70B) based on their performance.

Datasets

The study utilizes two verifiable datasets to ensure objective ground-truth verdicts:

  1. DeepScaleR: A dataset containing 40K Olympiad-style, reasoning-intensive math problems.
  2. MMLU-Pro: A multi-turn, knowledge-intensive dataset spanning 14 diverse domains (STEM, law, humanities, etc.).

By looking at the Pass@1 accuracy of various generators on these datasets, the authors found a clear capability gap that naturally separated the models into distinct weak and strong clusters.

Key Insights and Ideas

  • Judges don't automatically future-proof: If you train a judge on responses from weaker models, its performance drops when evaluating stronger models. It cannot naturally generalize to higher-quality reasoning.
  • DPO unlocks backward-compatibility: Interestingly, judges trained on stronger models are remarkably good at evaluating weaker models, often outperforming the judges that were actually trained on those weak models. This effect is particularly pronounced when using DPO training.
  • The Question Generalization Paradox: While DPO is superior for handling response distribution shifts, SFT actually yields better generalization to unseen questions. DPO-trained judges tend to overfit more to the specific question semantics seen during training.
  • Continual learning balances the scales: Instead of training a new judge from scratch every time a new generator drops, incrementally fine-tuning an existing judge via continual learning preserves backward-compatibility while adapting to future models.

Key Results

  • Future-Proofing Failure: Across all models and training recipes on DeepScaleR, FutureProof metrics were uniformly negative (ranging from -0.9 to -6.2 percentage points), showing clear performance degradation when facing stronger models.
  • The Refresh Advantage: Re-training judges on up-to-date, strong responses consistently improved performance. The largest gain was seen using DPO on Mistral-24B, yielding a 7.6 percentage point absolute improvement over the stale judge.
Future-proofing and Refresh Advantage metrics on DeepScaleR. All models show performance degradation on stronger responses (negative values in a), but benefit significantly from re-training (b).
  • Strong Backward-Compatibility: DPO-trained judges achieved positive BackCompatibility scores (up to +2.1 points), meaning a judge trained on newer models is a drop-in replacement that performs better on old models than the original judge did.
Backward-compatibility and Compatibility Shift metrics on DeepScaleR. DPO-based models show minimal drops or absolute gains when evaluating older responses.

Limitations and Future Work

The authors note a few constraints in their study:

  • Compute Limitations: Continual learning experiments were restricted to DPO-based training on the DeepScaleR dataset due to resource constraints.
  • Teacher Dependency: The training data relies on GPT-4o to generate the Chain-of-Thought (CoT) explanations used for distillation, meaning the judges are inherently bound by the teacher's reasoning quality.
  • Scope of Tasks: The study focuses heavily on objective, verifiable tasks (math and multiple-choice knowledge). How these distribution shifts affect subjective tasks like creative writing or roleplay remains an open question for future research.

Summary updated

Abstract

The LLM-as-a-judge paradigm is widely used in both evaluating free-text model responses and reward modeling for model alignment and fine-tuning. Recently, fine-tuning judges with judge-specific data has emerged as an often preferred choice over directly prompting frontier models as judges, as the former achieves better performance with smaller model sizes while being more robust to common biases. However, the standard evaluation ignores several practical concerns of fine-tuned judges regarding their real-world deployment. In this paper, we identify and formalize three aspects that affect the shelf life of these judges: future-proofing and backward-compatibility -- how well judges fine-tuned on responses by today's generator models perform on responses by future models or past models, as well as question generalization -- how well judges generalize to unseen questions at test time. We study these three aspects under a unified framework with varying train and test distributions in two reasoning datasets, three SFT- and DPO-based fine-tuning algorithms, and three different backbone models. Experiments suggest that future-proofing is challenging for most models, while backward-compatibility is relatively easy, with DPO-trained models consistently improving performance. We further find that continual learning provides a more balanced adaptation to shifts between older and newer response distributions than training solely on stronger or weaker responses. Moreover, all models exhibit some degree of performance degradation when moving from questions seen during training to unseen ones, showing that current judges do not fully generalize to unseen questions. These findings provide insights into practical considerations for developing and deploying judge models in the face of ever-changing generators.
BibTeX
@inproceedings{singh2026shelf,
  author = {Janvijay Singh and Austin Xu and Yilun Zhou and Yefan Zhou and Dilek Hakkani-Tur and Shafiq Joty},
  title = {{On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=fVTqNpny5r},
}
Figure