TL;DR
This paper introduces Latent-State Supervised Reinforcement Learning (LSRL), a novel method that applies reinforcement learning to the internal, hidden states of language models, rather than just the final output. By providing dense, step-by-step rewards for the model's internal reasoning process, LSRL significantly boosts mathematical reasoning accuracy without increasing computational cost at inference time.
What Problem Does This Paper Address?
Latent-recurrent language models (LR-LMs) like Huginn refine their understanding through iterative updates to internal "hidden states" rather than explicitly generating step-by-step reasoning chains (like chain-of-thought). While this approach can be more parameter-efficient, it suffers from a "sparse credit assignment" problem: traditional training methods only reward the final answer, making it hard for the model to learn from its intermediate reasoning steps. This opacity hinders their ability to perform complex mathematical reasoning tasks, leading to lower accuracy compared to models that explicitly show their work.
Methods
The core innovation is Latent-State Supervised Reinforcement Learning (LSRL), a variant of Guided Reward Policy Optimization (GRPO). Here's how it works:
- Process Supervision on Latent States: Instead of rewarding only the final answer, LSRL provides rewards at each latent step. This is achieved by decoding the model's internal hidden states at various depths into textual "snapshots" of its reasoning.
- GPT-4.1-nano Grader: A lightweight GPT-4.1-nano model is used to evaluate these intermediate snapshots. It assigns scores for "Internal Quality" (logical consistency) and "Mathematical Progress" (how much the step advances towards the solution).
- Efficient Caching: To avoid recomputing hidden states multiple times, a one-pass caching mechanism is employed. This allows all intermediate states to be generated during a single forward/backward pass.
- LoRA for Parameter Efficiency: To make fine-tuning feasible on limited hardware, the authors use Low-Rank Adaptation (LoRA) adapters. This significantly reduces the number of trainable parameters (by 99%), enabling single-GPU training.
- Combined Reward: The final reward is a weighted sum of the process rewards (from intermediate steps) and a binary bonus for the correct final answer.
Datasets
The models were trained and evaluated on three mathematical reasoning benchmarks:
- GSM-8K: Grade-school word problems.
- MATH: Theorem-style proofs (more complex).
- MathQA: Multi-step arithmetic questions.
Crucially, the reinforcement learning phase for LSRL was trained on a small subset of 500 GSM-8K problems, demonstrating the method's efficiency.
Key Insights and Ideas
- The Bottleneck is Sparse Credit Assignment: The paper strongly argues that the limited performance of LR-LMs on math tasks stems from the difficulty of assigning credit to intermediate steps. LSRL directly tackles this.
- Dense Rewards for Internal States Work: Providing rewards for the model's internal reasoning process, not just the final output, is highly effective for improving mathematical reasoning.
- Efficiency is Key: LSRL achieves significant accuracy gains while maintaining the same inference cost as the baseline models, and drastically reduces training compute compared to naive approaches. LoRA makes this fine-tuning practical.
- Recurrent Depth Matters: While LSRL shows promise, the paper suggests a minimum recurrent depth is necessary for process supervision to be effective. Shallow recurrences (r=4) yielded minimal gains.
Key Results
- Accuracy Improvements: LSRL improved accuracy by +4.27 points on GSM-8K, +1.33 points on MATH, and +2.06 points on MathQA compared to the supervised depth-8 baseline.
- Efficiency: LSRL-r8 achieved approximately 75% of the GSM-8K score of a depth-32 model while using only 25% of the recurrent compute.
- Training Efficiency: The method was fine-tuned on just 500 GSM-8K problems using a single NVIDIA L40S GPU.
- Qualitative Improvements: Figure 1 shows LSRL generating correct intermediate steps early on, while the baseline model drifts off-topic and makes calculation errors. LSRL's reasoning becomes more human-legible.
Limitations and Future Work
- Prompt Echo and Fluff: The model sometimes copies prompt fragments or includes irrelevant phrases in early reasoning steps.
- Unit/Type Confusion: Errors like miscalculating revenue based on planks instead of tables persist, suggesting limitations in the reward model's understanding of complex concepts.
- Limited Training Data: The RL phase used only 500 GSM-8K problems, which might limit the diversity of learned strategies.
- MATH Benchmark Challenges: The modest gains on the MATH benchmark are attributed to its inherent complexity (proofs, symbolic manipulation) and the reward model's focus on local algebraic progress.
- Future Directions:
- Scaling Recurrent Depth: Applying LSRL to deeper models (r=16, r=32) is expected to yield further gains.
- Specialized Reward Models: Developing symbol-aware reward models could better handle complex mathematical steps and reduce unit/type errors.
- Curriculum Learning: Using difficulty-aware curricula could accelerate learning and improve transfer.
- Cross-Domain Transfer: Investigating LSRL's effectiveness in domains beyond mathematics (e.g., coding, logical reasoning) is a promising avenue.
Summary updated