Intrinsic Reward Optimization via Internal Model Probabilities

112 concepts81 papers1 proposal

The Bottleneck of External Supervision

Reinforcement Learning with Verifiable Rewards (RLVR) has driven massive breakthroughs in large language model (LLM) reasoning. However, standard RLVR relies heavily on external grounding—such as code execution environments or mathematical truth values—limiting its application to domains with clear, programmatic verifiers. To expand reasoning capabilities to open-ended or unverifiable tasks, traditional approaches train Process Reward Models (PRMs) using dense, human-annotated step-by-step labels. This introduces a severe data bottleneck and exposes the system to reward hacking due to distribution mismatch as the target policy evolves.

To bypass this bottleneck, a powerful paradigm has emerged: utilizing the model's own internal probability space as a dense, self-supervised reward signal. Instead of relying on external judges or static reward models, the system leverages the model's implicit map of its own certainty to guide policy optimization and tree search.

The Internal Probability Space as a Reward Function

The core intuition behind this approach is that an LLM's internal token probabilities are highly calibrated with respect to correctness. As Audrey Huang et al. demonstrate, correct responses naturally possess higher internal log-probabilities than incorrect ones, even in the absence of external labels. This mathematical alignment allows a model's average log-likelihood to serve as a direct reward function, establishing a theoretical bridge between generative sequence probabilities and reinforcement learning values (Junzhuo Li et al.).

By treating internal model confidence as an intrinsic reward signal, models can self-improve through unsupervised reinforcement learning. For instance, Mihir Prabhudesai et al. and Shivam Agarwal et al. show that minimizing token-level entropy (maximizing confidence) during fine-tuning acts as a powerful unsupervised objective. This entropy minimization forces the model to produce deterministic, high-confidence reasoning chains, effectively eliciting latent problem-solving capabilities without requiring ground-truth datasets.

Deriving Dense Step-Wise Rewards Implicitly

While global confidence can optimize a full sequence, complex reasoning requires granular credit assignment—identifying exactly which step in a long chain of thought was useful or flawed. Intrinsic reward optimization achieves this by tracking how intermediate steps alter the model's internal certainty about the final outcome.

Instead of collecting expensive step-level human annotations, Lifan Yuan et al. show that step-level rewards can be extracted from outcome-only labels by parameterizing the reward function as the log-likelihood ratio between the current policy and a reference model. Because autoregressive models naturally decompose total response probability into a sequence of token-level conditional probabilities, the outcome reward mathematically breaks down into a sum of step-level rewards.

[Initial Prompt] 
       │
       ▼
[Step 1 Generated] ──► Measure internal confidence shift
       │
       ▼
[Step 2 Generated] ──► Measure internal confidence shift
       │
       ▼
[Final Answer]     ──► Evaluate outcome-level calibration

Similarly, Eunseop Yoon et al. introduce a mechanism that tracks the model's internal confidence at every step of a trajectory. By rewarding paths where the probability of the correct answer progressively ascends, the system performs precise credit assignment. Junnan Liu et al. further refine this by measuring how intermediate thoughts change the conditional likelihood of reaching a correct final answer, penalizing erratic reasoning paths and rewarding steady progress.

A major risk of relying purely on internal feedback is that a model might become "confidently incorrect," optimizing for highly certain but logically flawed paths (Xingwu Chen et al.). To mitigate this, recent frameworks integrate on-policy tree search directly into the training loop to ground the intrinsic rewards.

Zhenyu Hou et al. and Yinya Huang et al. demonstrate that models can generate their own process-level supervision by sampling multiple reasoning branches from a single intermediate step. By calculating the average success rate of these branches, the model estimates the long-term value of specific actions. This self-contained feedback loop ensures that the process-level reward is always grounded in the model's current capabilities, eliminating the need for an external reward model and drastically reducing the risk of distribution shift.

Key Foundational Work

Reinforced Question Rewriting for Conversational Question Answering (2022) — contributed 2 concept elements to this direction.
Reinforced Question Rewriting for Conversational Question Answering (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_self_supervised_process_supervision_325c64de,
  title = {self-supervised process supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-supervised-process-supervision/rdr_325c64de60014380b41db74e21b4514d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_self_supervised_process_supervision_325c64de_v1,
  title = {self-supervised process supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-supervised-process-supervision/rdr_325c64de60014380b41db74e21b4514d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake