HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment

Zhanyu Liu, Qingguo Hu, Ante Wang, Chenqing Liu, Zhishang Xiang, Hui Li, Delai Qiu, Jinsong Su

TL;DR

Reinforcement Learning with Verifiable Reward (RLVR) excels at training reasoning models, but it suffers from severe "entropy collapse"—a premature loss of exploration diversity—when data is scarce. This paper introduces HEAL (Hybrid-domain Entropy dynamics ALignment), a framework that selectively injects general-domain data and uses a novel reward mechanism to align fine-grained exploration patterns. Remarkably, with just 32 target-domain samples, HEAL matches or outperforms models trained on 1,000 samples.

What Problem Does This Paper Address?

Recent breakthroughs in reasoning-oriented Large Language Models (LLMs), such as OpenAI o1 and DeepSeek-R1, rely heavily on Reinforcement Learning with Verifiable Reward (RLVR). RLVR uses rule-based outcomes (like a binary correct/incorrect flag for a math problem) to guide the model without needing a complex, hackable reward model.

However, RLVR assumes you have thousands of high-quality training examples. In specialized fields like medicine or advanced physics, such data is incredibly scarce. When you try few-shot RLVR in these low-resource settings, the model quickly overfits to the few successful paths it finds. This triggers entropy collapse: the model's token-generation probabilities narrow drastically, it stops exploring alternative reasoning paths, and performance plateaus. Traditional fixes simply try to force entropy higher across the board, but this often destabilizes training or causes "entropy explosion," where the model starts outputting incoherent text.

Methods

The authors propose HEAL (Hybrid-domain Entropy dynamics ALignment) to safely restore exploration without breaking the model's focus. HEAL consists of two primary components:

  1. High-Value General-Domain Data Selection: Instead of flooding the training pipeline with massive general datasets, HEAL filters a general-domain dataset (like commonsense reasoning) down to a small, high-value subset. It scores samples based on a combination of Reasoning Uncertainty (avoiding questions that are too easy or too hard) and Exploratory Diversity (pockets of high token-level entropy).
  2. Entropy Dynamics Alignment (EDA): Rather than treating entropy as a single average number, EDA looks at trajectory-level entropy dynamics—the step-by-step sequence of entropy fluctuations as a model generates a response. EDA acts as a soft reward mechanism, rewarding target-domain trajectories that exhibit similar structural entropy variations to successful general-domain trajectories.
Figure 2: Overview of our proposed HEAL framework. Left: We incorporate a small set of high-value general- domain data into few-shot RLVR to promote diverse exploration and mitigate entropy collapse in the target domain. Right: Entropy Dynamics Alignment reward guides the policy by aligning the trajectory-level entropy dynamics of few-shot target-domain data with those of the selected general-domain data, thereby further encouraging controlled increases in entropy and more diverse exploratory behaviors.

Datasets

The framework was evaluated across four distinct target domains and one general domain:

  • Medicine: MedBullets and MedXpertQA.
  • Physics: WebInstruct and physics subsets of C-Eval.
  • Code: LiveCodeBench (v1–v4 for training, v5 for evaluation) and HumanEval Plus.
  • Math: DeepScaleR (used to sample the full-shot baseline), AMC23, Math500, Minerva, and Olympiad.
  • General Domain (Source): CommonsenseQA.

Key Insights and Ideas

  • Structure Over Magnitude: The core "aha moment" of the paper is that when a model explores matters just as much as how much it explores. Naive entropy maximization causes the model to lose its train of thought. By aligning the dynamics (the ups and downs of uncertainty during generation), HEAL teaches the model when to open up its search space and when to narrow down on a logical conclusion.
  • General Skills as an Exploration Template: General-domain data (like commonsense reasoning) doesn't provide domain-specific knowledge for a medical or coding task. Instead, it provides fundamental reasoning patterns. It acts as a safe sandbox where the model can maintain healthy exploration behaviors that transfer implicitly to the target domain.

Key Results

The authors evaluated HEAL using Qwen3 (1.7B and 4B) and LLaMA-3.2 (3B) models.

  • The 32-Sample Miracle: With only 32 target-domain samples, HEAL consistently matched or surpassed the "Full-shot" baseline trained on 1,000 target samples. For instance, Qwen3-4B-Base using HEAL achieved an average score of 46.99% in Physics and 53.42% in Code, outperforming its Full-shot counterparts (44.57% and 51.00%, respectively).
  • Defeating Existing Regularizers: HEAL significantly outperformed state-of-the-art entropy regularization techniques like 80/20 Strategy, Clip-Higher, and KL-Cov. As shown in the training curves, while standard Entropy Loss leads to an uncontrolled "entropy explosion," HEAL stabilizes token-level entropy at a healthy, productive level comparable to full-shot training.
Figure 3: Average token-level entropy during train- ing under different entropy regularization methods and HEAL. We conduct experiments on the Qwen3-1.7B- Base model using the same training dataset.

Limitations and Future Work

While promising, the authors note a couple of limitations:

  • Scalability: Due to computational constraints, HEAL was only validated on smaller base models (up to 8B parameters). Its efficacy on frontier-scale models (70B+) remains to be proven.
  • Overhead: Introducing a hybrid data pipeline and calculating trajectory-level similarity metrics introduces minor computational overhead during the training phase, though this is heavily offset by the massive reduction in required target data.

Summary updated

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective for training reasoning-oriented large language models, but existing methods largely assume high-resource settings with abundant training data. In low-resource scenarios, RLVR is prone to more severe entropy collapse, which substantially limits exploration and degrades reasoning performance. To address this issue, we propose Hybrid-domain Entropy dynamics ALignment (HEAL), a framework tailored for few-shot RLVR. HEAL first selectively incorporates high-value general-domain data to promote more diverse exploration. Then, we introduce Entropy Dynamics Alignment (EDA), a reward mechanism that aligns trajectory-level entropy dynamics between the target and general domains, capturing both entropy magnitude and fine-grained variation. Through this alignment, EDA not only further mitigates entropy collapse but also encourages the policy to acquire more diverse exploration behaviors from the general domain. Experiments across multiple domains show that HEAL consistently improves few-shot RLVR performance. Notably, using only 32 target-domain samples, HEAL matches or even surpasses full-shot RLVR trained with 1K target-domain samples.
BibTeX
@inproceedings{liu2026healing,
  author = {Zhanyu Liu and Qingguo Hu and Ante Wang and Chenqing Liu and Zhishang Xiang and Hui Li and Delai Qiu and Jinsong Su},
  title = {{HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1418},
}
Figure