In the world of Large Language Models (LLMs), we often use "Generative Reward Models" (GenRMs) to act as judges, deciding which of two AI responses is better. These judges don't just give a score; they provide a natural language rationale—a "why"—for their decision.
However, a correct verdict doesn't always mean a sound justification. A judge might pick the right answer based on hallucinated evidence or by ignoring the most important part of the prompt. This new paper introduces DeCert (Decision-Certificate Supervision), a framework designed to train these judges to be not just right, but right for the right reasons, all without needing expensive human-written reference explanations.
TL;DR
DeCert treats rationales as "decision certificates" that must be sound, comprehensive, and logically consistent with the final verdict. By supervising the quality of these justifications—even when the final verdict is already correct—DeCert significantly improves reward model accuracy, rationale quality, and robustness to positional bias.
What Problem Does This Paper Address?
When we train reward models, we usually focus on outcome supervision: did the model pick the same response a human would? While this works for simple tasks, it fails to distinguish between a well-reasoned judgment and one that "stumbled" onto the right answer.
Existing methods to fix this often rely on "reference rationales" (human-written explanations). These are incredibly costly to produce and don't necessarily prove that the model's own reasoning supports its own verdict. As shown in Figure 1, unreliable judges can suffer from "evidence hallucination" or "irresponsible deployment" where the stated rationale actually conflicts with the final output.
Methods
The authors propose a shift in perspective: a rationale should be viewed as a decision certificate. To be valid, this certificate must meet three criteria:
- Soundness ($S_{snd}$): Are the factual claims and local inferences valid? DeCert uses a Natural Language Inference (NLI) verifier to check if claims are grounded in the responses.
- Coverage ($S_{cov}$): Does the rationale address the "material issues"? DeCert uses an LLM-based analyzer to identify key requirements or errors in the input before looking at the rationale, then checks if the rationale covers them.
- Decision Support ($S_{dec}$): Does the verified evidence actually lead to the final verdict? It calculates whether the points awarded to Response A vs. Response B actually justify the "Winner: A" conclusion.
These three scores are combined using a geometric mean to produce a final Rationale Score ($Q_{proc}$).
The DeCert Training Pipeline
As illustrated in Figure 2, DeCert uses these scores to create a smarter training signal for Direct Preference Optimization (DPO):
- Outcome-based pairs: If one judgment is correct and another is wrong, the correct one is preferred.
- Rationale-based pairs: If both judgments reach the correct verdict, DeCert ranks them by their $Q_{proc}$ score. This forces the model to prefer the judgment with the better justification.
Datasets
- Training: The models were trained on a subset of HelpSteer3-Preference, consisting of roughly 12k instances across General, STEM, Code, and Multilingual tasks.
- Evaluation:
- HelpSteer3 (test set): In-distribution evaluation.
- RM-Bench: An out-of-distribution (OOD) benchmark for subtle differences and style biases.
- JudgeBench: A benchmark focused on knowledge-heavy and complex reasoning tasks.
- HelpSteer3-Atomic: Used to measure rationale quality against human checklists.
Key Insights and Ideas
- Supervising the "Correct" is Crucial: The most striking insight is that we can extract a powerful training signal from judgments that are already "correct." By comparing two correct verdicts and favoring the one with the better "certificate," the model learns a much deeper representation of what makes a good response.
- Reference-Free Auditing: You don't need a "gold" human explanation to know if a rationale is bad. If a rationale claims "Response A has no typos" but the text clearly shows a typo, the rationale is unsound. DeCert automates this "internal" auditing.
- Geometric Mean Matters: Using a geometric mean for the quality score ensures that a model can't "cheat" by having perfect coverage but zero soundness. It must perform well across all three dimensions to get a high score.
Key Results
DeCert consistently outperformed strong baselines, including GPT-4o and Claude-3.5-Sonnet (when used as judges), across multiple benchmarks.
- Accuracy Gains: On the HelpSteer3 test set, DeCert improved original-order accuracy by 9.1 points for Llama-3.1-8B and 6.3 points for Qwen3-8B.
- Positional Robustness: One of the biggest headaches in LLM evaluation is "positional bias" (the tendency to pick the first response). DeCert improved position-consistent accuracy (PCA) by up to 14.8 points without any explicit swap-augmentation during training.
- Rationale Quality: As shown in Figure 3, adding rationale preference pairs specifically boosted positional consistency. Furthermore, fine-grained analysis showed a massive reduction in "non sequiturs" (logic that doesn't follow), dropping from 21.5% to just 3.3%.
Limitations and Future Work
- Analyzer Dependency: DeCert relies on an LLM-based "material-issue analyzer." If the analyzer is biased or misses a key issue, the supervision signal will be flawed.
- Model Scale: The study focused on 8B parameter models. While results are promising, it remains to be seen how these techniques scale to frontier-class models (70B+).
- The "Black Box" Problem: While DeCert improves the generated rationale, it doesn't guarantee that this rationale perfectly reflects the model's internal "thought process" (latent decision process). Future work could explore aligning the two more closely.
Summary updated