TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) helps Large Language Models (LLMs) reason better, but popular algorithms like GRPO often suffer from "entropy collapse," where the model prematurely becomes deterministic and stops exploring. This paper introduces On-Policy Entropy Flow Optimization (OPEFO), a method that analyzes entropy at the token level and adaptively balances entropy-increasing and entropy-decreasing updates. OPEFO stabilizes training and boosts reasoning performance across multiple mathematical benchmarks while remaining strictly on-policy and computationally efficient.
What Problem Does This Paper Address?
When training LLMs with reinforcement learning using automated, verifiable rewards (like a compiler checking code or a regex checking a math answer), models often hit a wall known as entropy collapse. Early in training, the model's policy entropy drops sharply to near zero. This means the model becomes overconfident and deterministic, repeatedly exploiting a narrow set of high-reward answer patterns instead of exploring diverse reasoning paths.
Existing solutions fall short:
- Entropy Regularization: Adds a blunt penalty to encourage higher entropy, but it can cause uncontrolled entropy explosion later in training and destabilize optimization.
- Clipping Heuristics (e.g., Clip-higher): Relax importance-sampling bounds to keep more exploration tokens, but they rely on an outdated reference policy, violating the principles of strict on-policy training.
The authors set out to understand why entropy collapse happens at a granular level and how to fix it without breaking strict on-policy optimization.
Methods
The authors analyze training dynamics through a token-level entropy flow perspective. Using a first-order Taylor expansion, they estimate the entropy change ($\Delta H_t$) for individual tokens during a policy update. This allows them to split a batch of tokens into two distinct groups:
- Entropy-increasing tokens ($S^+$): Updates that broaden the model's predictive distribution (encouraging exploration).
- Entropy-decreasing tokens ($S^-$): Updates that sharpen the distribution (encouraging exploitation).
They discover that in standard GRPO training, entropy-decreasing updates overwhelmingly dominate the early stages, causing the collapse.
To fix this, OPEFO introduces a reweighted gradient objective using an adaptive balancing coefficient ($\lambda$). Instead of picking a static hyperparameter, OPEFO dynamically calculates an optimal $\lambda^*$ for each batch to satisfy a "zero net entropy flow" constraint:
$$\sum_{t \in S^+} (1 + \lambda)\Delta H_t + \sum_{t \in S^-} (1 - \lambda)\Delta H_t \approx 0$$
This rescales the gradients so that the forces of exploration and exploitation are perfectly balanced. Crucially, because it only rescales gradient components based on the current batch statistics, it requires no reference policy and remains strictly on-policy.
Datasets
- Training: DAPO-17K, a dataset containing high-quality reasoning trajectories annotated with verifiable rewards.
- Evaluation: Six prominent mathematical reasoning benchmarks: AIME24, AIME25, AMC23, MATH500, Minerva Math, and OlympiadBench.
Key Insights and Ideas
- The Root Cause of Collapse: Entropy collapse isn't a mysterious global phenomenon; it is driven by a severe token-level numerical imbalance early in training where sharpening updates consistently outweigh broadening updates.
- Why Clipping Works (and Fails): The authors show that heuristics like Clip-higher work because they accidentally preserve more entropy-increasing tokens by relaxing the upper clipping bound. However, because they rely on importance sampling ratios, they are fundamentally incompatible with clean, strict on-policy training.
- Strict On-Policy Benefits: Moving from approximate on-policy (multiple mini-batch updates per rollout) to strict on-policy (one update per rollout) provides a cleaner optimization signal and better baseline performance, but it does not inherently prevent entropy collapse without OPEFO.
Key Results
The authors evaluated OPEFO using two base models: Qwen2.5-Math-7B and Qwen3-Base-4B.
- Superior Accuracy: OPEFO achieved the best overall performance, reaching an average accuracy of 52.4% on Qwen2.5-Math-7B (outperforming standard GRPO at 45.5% and strict on-policy GRPO at 50.1%).
- Stable Training Dynamics: As shown in the training curves, OPEFO maintains a smooth, stable entropy trajectory throughout training, avoiding both the premature determinism of GRPO and the late-stage inflation of entropy regularization.
- Better Exploration: OPEFO consistently achieved higher
Pass@kmetrics (e.g., Pass@32 and Pass@64), proving that it helps the model discover a more diverse set of valid reasoning paths. - Zero Computational Overhead: Despite the extra token-level math, OPEFO's strict on-policy update strategy results in a per-batch runtime of 149 seconds, making it slightly faster than standard approximate on-policy GRPO (158 seconds) due to fewer sequential updates per rollout batch.
Limitations and Future Work
- First-Order Approximation: The token-level entropy estimation relies on a simplified independent softmax assumption, which abstracts away complex, higher-order token interactions inside large transformer architectures.
- Local vs. Global Optimum: The "zero net entropy flow" target is a local, batch-level heuristic that acts as a sufficient condition for stability, but it may not be globally optimal across all types of tasks.
- Scope of Rewards: The evaluation was restricted to sparse, verifiable rewards in mathematical reasoning. Testing OPEFO's efficacy under dense reward structures or more complex credit assignment scenarios remains an open direction for future research.
Summary updated