TL;DR
Reinforcement Learning with Verifiable Reward (RLVR) excels at training reasoning models, but it suffers from severe "entropy collapse"—a premature loss of exploration diversity—when data is scarce. This paper introduces HEAL (Hybrid-domain Entropy dynamics ALignment), a framework that selectively injects general-domain data and uses a novel reward mechanism to align fine-grained exploration patterns. Remarkably, with just 32 target-domain samples, HEAL matches or outperforms models trained on 1,000 samples.
What Problem Does This Paper Address?
Recent breakthroughs in reasoning-oriented Large Language Models (LLMs), such as OpenAI o1 and DeepSeek-R1, rely heavily on Reinforcement Learning with Verifiable Reward (RLVR). RLVR uses rule-based outcomes (like a binary correct/incorrect flag for a math problem) to guide the model without needing a complex, hackable reward model.
However, RLVR assumes you have thousands of high-quality training examples. In specialized fields like medicine or advanced physics, such data is incredibly scarce. When you try few-shot RLVR in these low-resource settings, the model quickly overfits to the few successful paths it finds. This triggers entropy collapse: the model's token-generation probabilities narrow drastically, it stops exploring alternative reasoning paths, and performance plateaus. Traditional fixes simply try to force entropy higher across the board, but this often destabilizes training or causes "entropy explosion," where the model starts outputting incoherent text.
Methods
The authors propose HEAL (Hybrid-domain Entropy dynamics ALignment) to safely restore exploration without breaking the model's focus. HEAL consists of two primary components:
- High-Value General-Domain Data Selection: Instead of flooding the training pipeline with massive general datasets, HEAL filters a general-domain dataset (like commonsense reasoning) down to a small, high-value subset. It scores samples based on a combination of Reasoning Uncertainty (avoiding questions that are too easy or too hard) and Exploratory Diversity (pockets of high token-level entropy).
- Entropy Dynamics Alignment (EDA): Rather than treating entropy as a single average number, EDA looks at trajectory-level entropy dynamics—the step-by-step sequence of entropy fluctuations as a model generates a response. EDA acts as a soft reward mechanism, rewarding target-domain trajectories that exhibit similar structural entropy variations to successful general-domain trajectories.
Datasets
The framework was evaluated across four distinct target domains and one general domain:
- Medicine: MedBullets and MedXpertQA.
- Physics: WebInstruct and physics subsets of C-Eval.
- Code: LiveCodeBench (v1–v4 for training, v5 for evaluation) and HumanEval Plus.
- Math: DeepScaleR (used to sample the full-shot baseline), AMC23, Math500, Minerva, and Olympiad.
- General Domain (Source): CommonsenseQA.
Key Insights and Ideas
- Structure Over Magnitude: The core "aha moment" of the paper is that when a model explores matters just as much as how much it explores. Naive entropy maximization causes the model to lose its train of thought. By aligning the dynamics (the ups and downs of uncertainty during generation), HEAL teaches the model when to open up its search space and when to narrow down on a logical conclusion.
- General Skills as an Exploration Template: General-domain data (like commonsense reasoning) doesn't provide domain-specific knowledge for a medical or coding task. Instead, it provides fundamental reasoning patterns. It acts as a safe sandbox where the model can maintain healthy exploration behaviors that transfer implicitly to the target domain.
Key Results
The authors evaluated HEAL using Qwen3 (1.7B and 4B) and LLaMA-3.2 (3B) models.
- The 32-Sample Miracle: With only 32 target-domain samples, HEAL consistently matched or surpassed the "Full-shot" baseline trained on 1,000 target samples. For instance, Qwen3-4B-Base using HEAL achieved an average score of 46.99% in Physics and 53.42% in Code, outperforming its Full-shot counterparts (44.57% and 51.00%, respectively).
- Defeating Existing Regularizers: HEAL significantly outperformed state-of-the-art entropy regularization techniques like 80/20 Strategy, Clip-Higher, and KL-Cov. As shown in the training curves, while standard Entropy Loss leads to an uncontrolled "entropy explosion," HEAL stabilizes token-level entropy at a healthy, productive level comparable to full-shot training.
Limitations and Future Work
While promising, the authors note a couple of limitations:
- Scalability: Due to computational constraints, HEAL was only validated on smaller base models (up to 8B parameters). Its efficacy on frontier-scale models (70B+) remains to be proven.
- Overhead: Introducing a hybrid data pipeline and calculating trajectory-level similarity metrics introduces minor computational overhead during the training phase, though this is heavily offset by the massive reduction in required target data.
Summary updated