Reinforcement Learning from Internal Confidence Signals

This direction is split into other directions: self-supervised process supervision. Its last version stays readable and citable here.

Version 1 of 190 papers111 conceptsfrozen 2026-10-04

Reinforcement Learning (RL) has traditionally relied on an external "judge" to provide feedback. In mathematics, this might be a code executor; in dialogue, it is often a human annotator. However, for many complex or open-ended tasks, external rewards are either too expensive to collect or impossible to define.

A growing body of research suggests that Large Language Models (LLMs) possess an "internal compass" that can substitute for these external signals. By rewarding a model for its own internal confidence—measured through the shape of its token probability distributions—researchers are finding that models can teach themselves to reason, simulate human behavior, and align with safety constraints without any external supervision.

The Entropy Bottleneck

The core intuition behind this research direction is that a model’s uncertainty is a signal of its reasoning quality. When a model is "guessing," its probability distribution across potential next tokens is flat (high entropy). When it has "figured out" a logical path, its distribution becomes peaked (low entropy).

Shivam Agarwal and colleagues demonstrated in The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning that simply rewarding a model for minimizing the entropy of its own predictions can boost reasoning performance to match models trained on tens of thousands of labeled examples. Similarly, the RENT framework by Mihir Prabhudesai et al. shows that maximizing confidence alone allows models to naturally become more accurate across math and science benchmarks.

This works because entropy minimization acts as a "sharpening" mechanism. As Audrey Huang and Cyril Zhang argue, self-improvement algorithms essentially force the model to prioritize high-likelihood, high-quality sequences it already knows how to verify, even if it couldn't consistently generate them initially.

Measuring Self-Certainty

To turn this intuition into a training signal, researchers use several mathematical proxies for "confidence":

  1. KL Divergence from Uniform: In the Shop-R1 framework, Tian Wang and Jiri Gesi reward the model based on how much its output distribution deviates from a uniform distribution. This encourages "decisive" internal reasoning before the model commits to an action.
  2. The Probability Gap: Milica Gasic introduced RLSF, which uses the gap between the top two predicted tokens as a reward. A large gap suggests the model has a clear "winner" in its reasoning, serving as a proxy for correctness.
  3. Token Concentration: The INTUITOR method by Xuandong Zhao and Sergey Levine uses the concentration of probability in token distributions to bypass the need for ground-truth answers in coding and math.

Rewarding the Path via the Destination

Another facet of this direction uses the model's confidence in a final answer to reward the intermediate reasoning steps. This solves the "credit assignment" problem: knowing which specific thought in a long chain was actually helpful.

In the NOVER framework, W. M. Liu and Yulan He use the model’s own perplexity on a ground-truth answer as a reward for the reasoning steps that preceded it. If a reasoning chain makes the final correct answer seem more "obvious" (lower perplexity) to the model, that chain is rewarded.

Similarly, Haolin Chen and the Salesforce AI team developed LaTRO, which treats reasoning as a latent variable. They reward reasoning paths that increase the conditional likelihood of the correct outcome, effectively turning the model into its own evaluator. Yuanfu Wang and Chao Yang extend this in Native Reasoning Training (NRT), where the model is rewarded for "resolving" its own internal uncertainty through its reasoning traces.

Test-Time Refinement

These internal signals are not just for offline training; they can also be used to "think" more effectively during inference.

  • Latent Thought Policy Optimization (LTPO): This method optimizes continuous "thought" vectors at test time by maximizing the model's predictive certainty through gradient-based updates, as explored in Thinking on the Fly.
  • Rational Metareasoning (RaM): C. Nicolò De Sabbata and Antoine Bosselut use internal confidence to estimate the "Value of Computation." If a reasoning step doesn't significantly increase the model's confidence in an answer, the model learns to skip it, reducing inference costs by up to 45%.
  • $\phi$-Decoding: Fangzhi Xu and Jun Liu use simulated future rollouts to see which paths lead to the highest internal confidence, allowing the model to "look ahead" and avoid logical dead-ends.

Beyond Reasoning: Safety and Vision

The utility of internal confidence extends beyond logic puzzles. In Safety Instincts, Guobin Shen and Yi Zeng show that LLMs can defend themselves against jailbreaks by trusting their "internal compass"—rewarding themselves for staying in high-confidence (safe) states.

In the visual domain, Jiaxi Liu and Nanyang Ye use Anchor-Final Self-Supervision (AFS) to reduce hallucinations in vision-language models. By rewarding consistency between intermediate visual grounding and final predictions, the model learns to trust its internal visual evidence over linguistic biases.

By shifting the reward signal from external labels to internal consistency, this research direction suggests a path toward models that are not just better at following instructions, but are fundamentally more self-aware and computationally efficient.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reinforcement_learning_from_internal_con_18f7ba92,
  title = {Reinforcement Learning from Internal Confidence Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reinforcement-learning-from-internal-confidence-signals/rdr_18f7ba92db0342858c40723eb5d3bca9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reinforcement_learning_from_internal_con_18f7ba92_v1,
  title = {Reinforcement Learning from Internal Confidence Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reinforcement-learning-from-internal-confidence-signals/rdr_18f7ba92db0342858c40723eb5d3bca9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}