Learning from Saturated Data: Signals Beyond Correctness for LLM Training

Hanno Hiss, Jasper Dekoninck, Martin Vechev

TL;DR

As Large Language Models (LLMs) become more capable, they often "saturate" training datasets, achieving 100% accuracy and leaving no "incorrect" examples to learn from. This paper demonstrates that models can still improve by training on these saturated datasets using fine-grained quality signals—like token-level entropy and self-judgments—to distinguish between "good" and "better" correct answers.

What Problem Does This Paper Address?

In traditional LLM post-training (like RLHF or DPO), models learn by comparing correct answers to incorrect ones. However, we are entering an era of "benchmark saturation," where a model might already solve every question in a training set correctly. When every sampled response is "right," binary correctness provides zero learning signal.

The authors ask: Can we extract a learning signal from questions the model already solves with perfect empirical accuracy? They investigate whether "easy-to-hard" transfer is possible—specifically, whether refining the way a model solves easy, saturated problems can actually help it solve much harder problems it currently fails.

Methods

The researchers propose moving beyond binary "correct/incorrect" labels to two fine-grained quality signals:

  1. LLM-as-a-Self-Judge: The model performs pairwise comparisons of its own correct completions, essentially "voting" on which reasoning path is clearer or more efficient.
  2. Self-Uncertainty (Inverse Entropy): The model calculates the average token-level entropy of its output. High entropy suggests the model is "unsure" or "guessing" even if it gets the right answer. By training on low-entropy (high-confidence) completions, the model reinforces its most stable reasoning paths.

These signals are integrated into two training frameworks:

  • DPO (Direct Preference Optimization): A standard method for preference learning.
  • $\sigma$-RRHF (Logistic-Weighted RRHF): A novel variant of Rank Responses to Align Language Models with Human Feedback. It uses a logistic sigmoid function to weight the loss based on the magnitude of the quality difference between two solutions, focusing the model's attention on clear quality gaps.

Datasets

The study evaluates these methods on two primary reasoning tasks:

  • Chain Sum (from ReasoningGym): A synthetic arithmetic task where difficulty is precisely controlled by the number of terms and digits. This allows the authors to create a "strictly easier" training set to test transfer to harder problems.
  • GSM8K: A standard benchmark of grade-school math word problems, representing noisier, free-form reasoning.

Key Insights and Ideas

  • The "Model Compatibility" Hypothesis: Surprisingly, inverse entropy (a judge-free signal) often outperformed self-judging. The authors suggest this is because low-entropy completions represent solutions the model can already represent "stably." Reinforcing these helps the model internalize robust computational patterns.
  • Easy-to-Hard Transfer is Real: The model showed significant gains on hard problems even when trained exclusively on "saturated" easy problems. This suggests that the style and confidence of reasoning on simple tasks are foundational for tackling complex ones.
  • The Scorer Reliability Bottleneck: On complex tasks like GSM8K, the model's self-judgments often "anti-correlated" with a strong external judge (Qwen3-30B). If a model is a poor judge of its own quality, self-improvement training can actually degrade performance.
  • Instruction-Tuning Limits Entropy Gains: While entropy signals worked wonders for "Base" models, they were less effective (or even harmful) for instruction-tuned models. This is likely because instruction-tuning already aggressively lowers entropy, leaving little room for further optimization without "over-fitting" to the proxy.

Key Results

  • Chain Sum Success: On the arithmetic task, $\sigma$-RRHF using inverse entropy improved the model's accuracy (pass@1) by 18.6% over the base model, significantly outperforming standard Supervised Fine-Tuning (SFT).
  • GSM8K Gains: On the more complex GSM8K, the gains were more modest. Using inverse entropy with DPO provided a +3.3% boost in accuracy.
  • Self-Judge Failure: On GSM8K, training with self-judgments actually caused performance to drop below the base model, highlighting that "self-improvement" is only as good as the model's internal evaluator.

Limitations and Future Work

The authors acknowledge several constraints:

  • Model Scale: The experiments were primarily conducted on the Qwen3-1.7B-Base model. It remains to be seen if these dynamics shift significantly at the 70B+ parameter scale.
  • Metric Diversity: The study focused on only two quality metrics. Future work could explore rubric-based rewards or more sophisticated "process" rewards.
  • Task Scope: The findings were most robust in the constrained domain of arithmetic. Generalizing these signals to creative writing or open-ended coding remains an open challenge.
  • Offline vs. Online: The study used offline training. Integrating these quality signals into "online" Reinforcement Learning (like PPO or GRPO) could provide a even stronger, albeit more unstable, learning signal.

Summary updated

Abstract

The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate whether questions solved with perfect empirical accuracy can nevertheless be used to improve downstream performance. To do so, we replace binary correctness with two sources of a more fine-grained quality signal: (1) pairwise LLM self-judgments, in which the model evaluates the relative quality of its own solutions, and (2) token-level entropy, where token-level uncertainty is used as a proxy for solution quality. We incorporate these signals into several training algorithms and evaluate them on Qwen3-1.7B-Base. When training exclusively on a simple arithmetic task, quality-based signals improve performance by up to 18.6% over the base model, substantially outperforming rejection sampling. On GSM8K, however, gains are more modest and depend strongly on the quality signal. For instance, self-judgments show poor agreement with a stronger external judge and can even degrade performance below the base model. Overall, our results suggest that quality-based training can extract useful signal from saturated questions for base models, but that applying such signals to more complex tasks requires careful calibration and further study.
BibTeX
@inproceedings{hiss2026learning,
  author = {Hanno Hiss and Jasper Dekoninck and Martin Vechev},
  title = {{Learning from Saturated Data: Signals Beyond Correctness for LLM Training}},
  booktitle = {AI4Math @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=8WsT8YNJYo},
}
Figure