DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Xiaofan Li, Ming Yang, Zhiyuan Ma, Shichao Ma, Jintao Du, Yu Cheng, Weiqiang Wang, Zhizhong Zhang, Xin Tan, Yanyun Qu, Lizhuang Ma, Yuan XiearXiv 2026 · arXiv

TL;DR

DiPO (Disentangled Perplexity Policy Optimization) introduces a fine-grained approach to managing the exploration-exploitation trade-off in reinforcement learning for LLMs. By partitioning the sample space based on perplexity and correctness, it selectively injects learning signals into "dead zones"—groups of samples that are entirely correct or entirely incorrect—where standard algorithms suffer from vanishing gradients. This targeted intervention yields significant performance boosts in complex tasks like mathematical reasoning and multi-turn function calling.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR)—exemplified by algorithms like Group Relative Policy Optimization (GRPO)—has become a cornerstone for enhancing LLM reasoning. By checking answers against a ground truth (e.g., programmatic execution or math rule-checkers), models learn to optimize their generation outputs.

However, RLVR algorithms face a critical Exploration-Exploitation Trade-Off (EETO) dilemma:

  1. Degradation of Extreme Samples: As training progresses, large portions of sample groups fall into extreme categories: either "hard" groups (where every generated response is wrong) or "easy" groups (where every response is correct). Because GRPO calculates advantages relative to the group, uniform rewards result in an advantage of zero. This causes gradients to vanish, leaving the model with no signal to explore better paths for hard problems or stabilize success on easy ones.
  2. Ineffective Exploration/Exploitation: Perplexity (PPL) indicates whether a model is exploring creative paths (high PPL) or exploiting known patterns (low PPL). Standard training often results in "ineffective" states—such as error samples generated with low perplexity (stubbornly confident mistakes) or correct samples with high perplexity (lucky guesses).
Figure 1: (a) The proportion of Easy/Normal/Hard groups in each step during the DAPO training. (b) The PPL distribution of correct and error samples in the validation set at 300th steps of DAPO training. (c) Illustration of four samples after PSD fine-grained partitioning.

Methods

The authors propose DiPO, which introduces two core mechanisms to resolve these dilemmas without destabilizing the underlying reward landscape:

1. Perplexity Space Disentangling (PSD)

Instead of applying a blanket exploration bonus, DiPO maintains a rolling queue of recent sample perplexities and correctness scores. It dynamically computes an optimal perplexity threshold ($\tau^*$) to divide the sample space into four distinct quadrants:

  • CL / EH (Correct-Low PPL / Error-High PPL): Naturally aligned states.
  • CH (Correct-High PPL): Unstable correct answers that need to be steered toward exploitation.
  • EL (Error-Low PPL): Confident mistakes that need to be forced into exploration.

2. Bidirectional Reward Reallocation (BRR)

To avoid the chaotic noise of directly adding perplexity values to the reward function, DiPO leaves "normal" mixed groups alone and intervenes only in the zero-gradient easy and hard groups:

  • For Hard Groups in the Exploitation Space (Low PPL): It identifies the response with the maximum perplexity and artificially sets its reward to 1, forcing the model to shift its policy toward higher-entropy exploration.
  • For Easy Groups in the Exploration Space (High PPL): It takes the response with the maximum perplexity and drops its reward to 0, penalizing unnecessary variance and encouraging stable exploitation.
Figure 2: Illustration of DiPO, consisting of three modules: PPL Queue, Perplexity Space Disentan- gling (PSD), and Bidirectional Reward Reallocation (BRR). Specifically, the PPL Queue caches PPL items; PSD is used for fine-grained sample partition; and then BRR performs reward allocation.

Datasets

The framework was validated across several demanding benchmarks:

  • Mathematical Reasoning: Trained on the DAPO-17K dataset and evaluated across six standard benchmarks: AIME24, AIME25, MATH500, AMC23, OlympiadBench (OLY), and Minerva (MIN).
  • Function Calling: Evaluated using the ToolRL framework on the BFCLv3 (Berkeley Function Calling Leaderboard) benchmark.
  • Risk Prediction: A real-world public opinion risk classification dataset consisting of 4,000 data points.

Key Insights and Ideas

  • Intervening Only in the Dead Zones: DiPO's most elegant trait is its surgical precision. By only reallocating rewards within groups where the variance is zero, it leaves the primary verification-driven gradients unperturbed for normal groups.
  • Perplexity as a Boundary, Not a Bias: Rather than shaping rewards continuously using raw perplexity values (which introduces massive variance), DiPO uses perplexity purely as a sorting mechanism to reallocate binary rewards. This preserves the stability of the original reward distribution.
  • High Coefficient Robustness: Unlike standard entropy-loss methods, which cause catastrophic performance collapse if their hyperparameters are slightly misconfigured, DiPO is highly forgiving. It maintains stable performance across a tenfold change in its loss weight ($\alpha$).

Key Results

DiPO consistently outperformed strong baselines like GRPO, DAPO, and Curiosity-Driven Exploration (CDE) across multiple model families (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, and Llama3.1-8B).

Mathematical Reasoning

On the Qwen3-8B-Base model, DiPO achieved the highest average accuracy across all six math benchmarks (54.79%), notably outperforming the standard DAPO baseline (53.23%) and showing strong capabilities on ultra-hard benchmarks like AIME24 (35.00% vs DAPO's 30.08%).

Table 1: Comparison of mathematical reasoning in ACC/mean@8 on 6 mathematics benchmarks. The best and second-best results are respectively marked in bold and underlined.

Function Calling & Risk Prediction

  • On BFCLv3 using Qwen2.5-7B-Instruct, DiPO pushed overall accuracy to 62.51% (up from ToolRL+DAPO's 61.06%). Crucially, its Multi-Turn Accuracy jumped from 19.75% to 24.50%, showing that fine-grained exploration prevents the model from getting stuck in multi-step dead ends.
  • In the risk prediction task, DiPO achieved a massive boost in Recall (79.49%) compared to the base model, ensuring critical risks weren't missed during automated monitoring.

Limitations and Future Work

  • Idealized Theoretical Assumptions: The mathematical proofs demonstrating that maximum-PPL adjustments shift policy entropy rely on idealized assumptions (such as global distribution statistics remaining invariant across samples). The authors note that strict mathematical proofs are unrealizable in vast LLM parameter spaces and require deep empirical validation.
  • Dependence on Verifiable Environments: DiPO relies on the presence of clear, binary, or structured verification rewards (like compiler outputs or math ground truths). Adapting this fine-grained EETO framework to open-ended generation tasks (like creative writing or abstract summarization) remains an open question.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.
BibTeX
@misc{li2026dipo,
  author = {Xiaofan Li and Ming Yang and Zhiyuan Ma and Shichao Ma and Jintao Du and Yu Cheng and Weiqiang Wang and Zhizhong Zhang and Xin Tan and Yanyun Qu and Lizhuang Ma and Yuan Xie},
  title = {{DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off}},
  year = {2026},
  eprint = {2604.13902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.13902},
}
Figure