Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin SongarXiv 2026 · arXiv

In the world of Large Language Models (LLMs), we often use "Generative Reward Models" (GenRMs) to act as judges, deciding which of two AI responses is better. These judges don't just give a score; they provide a natural language rationale—a "why"—for their decision.

However, a correct verdict doesn't always mean a sound justification. A judge might pick the right answer based on hallucinated evidence or by ignoring the most important part of the prompt. This new paper introduces DeCert (Decision-Certificate Supervision), a framework designed to train these judges to be not just right, but right for the right reasons, all without needing expensive human-written reference explanations.

TL;DR

DeCert treats rationales as "decision certificates" that must be sound, comprehensive, and logically consistent with the final verdict. By supervising the quality of these justifications—even when the final verdict is already correct—DeCert significantly improves reward model accuracy, rationale quality, and robustness to positional bias.

What Problem Does This Paper Address?

When we train reward models, we usually focus on outcome supervision: did the model pick the same response a human would? While this works for simple tasks, it fails to distinguish between a well-reasoned judgment and one that "stumbled" onto the right answer.

Existing methods to fix this often rely on "reference rationales" (human-written explanations). These are incredibly costly to produce and don't necessarily prove that the model's own reasoning supports its own verdict. As shown in Figure 1, unreliable judges can suffer from "evidence hallucination" or "irresponsible deployment" where the stated rationale actually conflicts with the final output.

Illustrations of risks arising from unreliable judging in GenRMs.

Methods

The authors propose a shift in perspective: a rationale should be viewed as a decision certificate. To be valid, this certificate must meet three criteria:

  1. Soundness ($S_{snd}$): Are the factual claims and local inferences valid? DeCert uses a Natural Language Inference (NLI) verifier to check if claims are grounded in the responses.
  2. Coverage ($S_{cov}$): Does the rationale address the "material issues"? DeCert uses an LLM-based analyzer to identify key requirements or errors in the input before looking at the rationale, then checks if the rationale covers them.
  3. Decision Support ($S_{dec}$): Does the verified evidence actually lead to the final verdict? It calculates whether the points awarded to Response A vs. Response B actually justify the "Winner: A" conclusion.

These three scores are combined using a geometric mean to produce a final Rationale Score ($Q_{proc}$).

The DeCert Training Pipeline

As illustrated in Figure 2, DeCert uses these scores to create a smarter training signal for Direct Preference Optimization (DPO):

  • Outcome-based pairs: If one judgment is correct and another is wrong, the correct one is preferred.
  • Rationale-based pairs: If both judgments reach the correct verdict, DeCert ranks them by their $Q_{proc}$ score. This forces the model to prefer the judgment with the better justification.
Overview of the DeCert framework.

Datasets

  • Training: The models were trained on a subset of HelpSteer3-Preference, consisting of roughly 12k instances across General, STEM, Code, and Multilingual tasks.
  • Evaluation:
    • HelpSteer3 (test set): In-distribution evaluation.
  • RM-Bench: An out-of-distribution (OOD) benchmark for subtle differences and style biases.
  • JudgeBench: A benchmark focused on knowledge-heavy and complex reasoning tasks.
  • HelpSteer3-Atomic: Used to measure rationale quality against human checklists.

Key Insights and Ideas

  • Supervising the "Correct" is Crucial: The most striking insight is that we can extract a powerful training signal from judgments that are already "correct." By comparing two correct verdicts and favoring the one with the better "certificate," the model learns a much deeper representation of what makes a good response.
  • Reference-Free Auditing: You don't need a "gold" human explanation to know if a rationale is bad. If a rationale claims "Response A has no typos" but the text clearly shows a typo, the rationale is unsound. DeCert automates this "internal" auditing.
  • Geometric Mean Matters: Using a geometric mean for the quality score ensures that a model can't "cheat" by having perfect coverage but zero soundness. It must perform well across all three dimensions to get a high score.

Key Results

DeCert consistently outperformed strong baselines, including GPT-4o and Claude-3.5-Sonnet (when used as judges), across multiple benchmarks.

  • Accuracy Gains: On the HelpSteer3 test set, DeCert improved original-order accuracy by 9.1 points for Llama-3.1-8B and 6.3 points for Qwen3-8B.
  • Positional Robustness: One of the biggest headaches in LLM evaluation is "positional bias" (the tendency to pick the first response). DeCert improved position-consistent accuracy (PCA) by up to 14.8 points without any explicit swap-augmentation during training.
  • Rationale Quality: As shown in Figure 3, adding rationale preference pairs specifically boosted positional consistency. Furthermore, fine-grained analysis showed a massive reduction in "non sequiturs" (logic that doesn't follow), dropping from 21.5% to just 3.3%.
Effect of rationale preference pairs on positional consistent accuracy.

Limitations and Future Work

  • Analyzer Dependency: DeCert relies on an LLM-based "material-issue analyzer." If the analyzer is biased or misses a key issue, the supervision signal will be flawed.
  • Model Scale: The study focused on 8B parameter models. While results are promising, it remains to be seen how these techniques scale to frontier-class models (70B+).
  • The "Black Box" Problem: While DeCert improves the generated rationale, it doesn't guarantee that this rationale perfectly reflects the model's internal "thought process" (latent decision process). Future work could explore aligning the two more closely.

Summary updated

Abstract

Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.
BibTeX
@misc{sun2026judging,
  author = {Guiquan Sun and Bing Bai and Martin Renqiang Min and Dongjin Song},
  title = {{Judging the Justification: Decision Certificate Supervision for Generative Reward Models}},
  year = {2026},
  eprint = {2610.27528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.27528},
}
Figure