In traditional reinforcement learning for language models, problem-solving and evaluation are treated as distinct tasks handled by separate models. An "actor" or policy network generates a solution, while an external "critic" or reward model scores its correctness.
However, a growing body of research suggests that for complex reasoning tasks, generation and verification are deeply intertwined. Splitting them introduces severe inefficiencies and vulnerabilities. Instead, a new paradigm unifies these capabilities within a single model or a tightly coupled training loop. By training a model to be both the solver and the verifier, we can fundamentally improve its internal representations, eliminate distribution shifts, and unlock autonomous self-correction.
The Problem with Isolated Critics
Why do separate reward models struggle when training language models to reason? The core issue is distribution shift.
As a generator undergoes reinforcement learning, its policy specializes and changes rapidly. A static, external reward model quickly finds itself evaluating text that looks entirely different from its training data. It struggles to accurately judge the highly specific outputs of the evolving generator.
To solve this, Fuxiang Zhang et al. demonstrate in Incentivizing LLMs to Self-Verify Their Answers that training a language model to simultaneously generate solutions and verify them ensures the verification process remains perfectly aligned with the model's own output distribution. Because the policy and the verifier share the same underlying model, they evolve together.
This co-evolution is a core mechanism behind the PAG framework introduced by Yuhua Jiang et al. in PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier. By managing both roles within a unified reinforcement learning process, improvements in generation naturally sharpen the model's ability to critique its own work, preventing the performance drop-offs typical of isolated systems.
Representation Synergy: Two Sides of the Same Coin
Beyond avoiding distribution shift, joint training works because the cognitive architectures of solving and verifying are mutually reinforcing. The internal representations required to generate a logical step-by-step solution are fundamentally similar to those required to audit it.
In Self-Verifying Reflection Helps Transformers with CoT Reasoning, Zhongwei Yu et al. show that forcing a model to output both reasoning steps and their corresponding verification labels acts as a powerful regularizer. This multi-task objective enhances the quality of the model's underlying embeddings. Remarkably, this structural improvement benefits forward planning and generative accuracy even when the explicit verification mechanism is completely disabled during inference.
This cross-task capability transfer operates on a simple principle: a model capable of generating a correct solution already possesses the latent understanding necessary to evaluate solutions. As Bei Li et al. explore in Libra: Assessing and Improving Reward Model by Learning to Think, leveraging a model's inherent problem-solving capacity directly elevates its precision as a judge.
Generative Verification: Letting the Critic Reason
Traditional verifiers act as discriminative classifiers, mapping a solution string to a single scalar or a binary token. This wastes the primary strength of a large language model: next-token text generation.
In Generative Verifiers: Reward Modeling as Next-Token Prediction, Lunjun Zhang et al. propose GenRM, framing verification as a generative task. By allowing the verifier to use Chain-of-Thought (CoT) to "reason" about why a solution is right or wrong before outputting a final judgment, the verifier achieves vastly superior accuracy over standard discriminative models.
Standard Discriminative Verifier:
[Solution] ---> [Binary Classifier] ---> Correct (0.87)
Generative Verifier (GenRM):
[Solution] ---> "Let's check the derivative in step 3...
The power rule was applied correctly, but
the constant was dropped incorrectly. Therefore..." ---> Incorrect
This generative approach is integrated directly into reinforcement learning loops by Kusha Sareen et al. in Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers ($RL^V$). By adding a supervised fine-tuning loss for verification to the standard RL reasoning objective, they train a single model to seamlessly alternate between solving a problem and outputting a concise "Yes/No" verification token.
To optimize this process, Jianyuan Zhong et al. demonstrate in Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifier that training generative verifiers with group-based policy optimization (such as GRPO) yields highly precise error detection and concise judgments, drastically reducing the token overhead typically required during inference-time scaling.
Closing the Loop with Self-Correction
When a single model can reliably evaluate its own output, it gains the ability to autonomously repair its mistakes in multi-turn interactions.
For example, the RISE framework introduced by Xiaoyuan Liu et al. in Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards embeds problem-solving and self-critique into a single training loop. The model generates multiple candidate solutions, scores them, and learns to identify its own logical errors.
This loop can be augmented with external environments. In Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning, Jiaqi Liu et al. align task-solving and evaluation distributions by integrating tools like Python code sandboxes into the loop.
To scale these training environments without relying on scarce human annotations, the authors of ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models (Andre Wang He et al.) show that synthesizing programmatic verifiers is far more effective than synthesizing raw solutions. Programmatic verifiers provide a highly stable, less error-prone reward signal that helps bootstrap the model's internal verification capabilities.
The Faithfulness Gap
While joint generation-verification training forces a model's internal states to align with ground-truth verification goals, a critical nuance remains: internal alignment does not guarantee external honesty.
In Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers, Vatsal Ananthula et al. study "verifier-coupled reasoning." They show that while adding a consistency loss successfully forces a model's internal hidden states to encode verifier information (making the verifier's output decodable from the weights), the model's generated natural language explanations can still diverge from its actual internal reasoning process.
This reveals that while joint training is a powerful diagnostic and optimization tool for shaping representations, closing the gap between internal verification states and faithful natural language explanations remains an open challenge.
Key Foundational Work
Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers (2025) — contributed 3 concept elements to this direction.
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier (2026) — contributed 3 concept elements to this direction.
Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers (2026) — contributed 3 concept elements to this direction.