Near-Future Policy Optimization

Chuanyu Qin, Chenxu Yang, Qingyi Si, Naibin Gu, Dingyu Yao, Zheng Lin, Peng Fu, Nan Duan, Jiaqi WangarXiv 2026 · arXiv

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) often struggles with sparse rewards early in training and performance plateaus later on. Near-Future Policy Optimization (NPO) elegantly solves this by letting a model learn from its own "near-future self"—a slightly more optimized checkpoint from the same training run. This approach perfectly balances trajectory quality against distribution shift, accelerating training convergence by ~2.1× and significantly raising the performance ceiling on complex reasoning tasks.

What Problem Does This Paper Address?

Post-training large language models using RLVR (such as GRPO or PPO with rule-based verifiers) has driven massive breakthroughs in AI reasoning. However, standard on-policy RLVR hits two structural walls:

  1. The Cold-Start Problem: Early in training, the model rarely stumbles upon the correct reasoning path, leading to sparse rewards and gradient collapse.
  2. The Late-Stage Plateau: Later in training, the model's exploration distribution narrows, causing it to plateau before fully mastering the task space.

To fix this, mixed-policy methods inject high-quality off-policy trajectories. However, they face a harsh trade-off between signal quality ($Q$) and variance cost ($V$). External teachers (like GPT-4) provide high-quality data but are distributionally far from the model, causing explosive gradient variance. Conversely, replaying past checkpoints keeps variance low but provides low-quality data capped by the model's own history. The authors formalize this tension as maximizing the effective learning signal:

$$\mathcal{S} = \frac{Q}{V}$$

Existing methods sit at suboptimal corners of this trade-off. This paper introduces a way to find the optimal sweet spot.

Methods

The authors introduce Near-Future Policy Optimization (NPO). Instead of looking outward to external teachers or backward to past checkpoints, NPO looks forward to a near-future checkpoint ($\Delta$ steps ahead) along the exact same optimization path. Because this future self shares the same architecture, initialization, and history, it remains distributionally close (low $V$), yet further optimization makes it smarter (high $Q$).

The Quality-Proximity Trade-off showing how NPO targets the high-S region compared to external teachers or past replay.

The authors validate NPO through two paradigms:

  1. Manual Interventions:
  2. Early-Stage Bootstrapping: Running a short "scout run" ahead of time to get a slightly advanced checkpoint that guides the main run through the initial sparse-reward minefield.
  3. Late-Stage Plateau Breakthrough: Continuing a plateaued run a bit further to obtain a stronger guide, then rolling back and replaying the plateaued segment with this future guidance.
  4. AutoNPO: An automated, online controller that monitors standard training logs for signs of exploration collapse (stagnating rewards and dropping entropy). When triggered, it empirically estimates the optimal checkpoint distance $\Delta^*$ that maximizes $\mathcal{S}(\Delta)$, automatically rolls back training, and injects future-self trajectories into the rollout groups.

Crucially, because the future self is so close to the current policy, the standard importance-sampling (IS) correction can be safely omitted without destabilizing training, saving substantial compute and memory.

Datasets

  • Training: MMFineReason-123K, a challenging multimodal dataset filtered to keep only the prompts where a strong baseline model (Qwen3-VL-4B-Thinking) failed every single rollout attempt.
  • Evaluation: Eight prominent multimodal reasoning benchmarks covering visual math, broad subject knowledge, and fine-grained visual discrimination: MMMU-Pro, MathVista, MathVision, WeMath, MathVerse, MMBench, MM-Star, and ZeroBench.

Key Insights and Ideas

  • The Temporal Sweet Spot: The effective learning signal $\mathcal{S}(\Delta)$ follows a distinct inverted U-shape. If the future checkpoint is too close, it offers no new knowledge; if it is too far, parameter drift causes variance to overwhelm the quality gains. NPO successfully operates right at the peak of this curve.
  • Entropy Re-expansion: Standard GRPO training suffers from steady entropy collapse as the model narrows down to a limited set of reasoning templates. AutoNPO’s interventions actively re-expand generation entropy, reopening exploration and preventing premature optimization collapse.
  • IS Correction is Omittable: One of the most surprising practical findings is that exact importance-sampling correction is marginal for NPO. Because the near-future guide is so close to the current policy, omitting IS doesn't cause training to collapse (unlike with external teachers), making NPO highly efficient.

Key Results

  • Performance Boost: Evaluated on Qwen3-VL-8B-Instruct, vanilla GRPO achieves an average accuracy of 60.25% across the eight benchmarks. Manual NPO improves this to 62.84%, and AutoNPO pushes it to 63.15%, setting a new state-of-the-art ceiling and outperforming strong baselines like RLEP and ExGRPO.
  • Training Speedup: NPO's early-stage intervention delivers a ~2.1× speedup in convergence during the initial training phase.
Validation accuracy during training showing the 2.1x speedup and higher ceiling achieved by NPO and AutoNPO.

Limitations and Future Work

  • While NPO works exceptionally well at the sequence level inside standard RLVR rollout groups, the authors note that alternative mechanisms—such as token-level On-Policy Distillation from the near-future self—remain unexplored.
  • This work represents the "temporal self" dimension of a broader conceptual framework the authors call Self-Taught RLVR. Future work will focus on introducing a "parallel self" to complement the "informed self" and "temporal self" paradigms.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a core post-training recipe. Introducing suitable off-policy trajectories into on-policy exploration accelerates RLVR convergence and raises the performance ceiling, yet finding a source of such trajectories remains the key challenge. Existing mixed-policy methods either import trajectories from external teachers (high-quality but distributionally far) or replay past training trajectories (close but capped in quality), and neither simultaneously satisfies the strong enough (higher $Q$ , more new knowledge to learn) and close enough (lower $V$ , more readily absorbed) conditions required to maximize the effective learning signal $\mathcal{S} = Q/V$. We propose \textbf{N}ear-Future \textbf{P}olicy \textbf{O}ptimization (\textbf{NPO}), a simple mixed-policy scheme that learns from a policy's own near-future self: a later checkpoint from the same training run is a natural source of auxiliary trajectories that is both stronger than the current policy and closer than any external source, directly balancing trajectory quality against variance cost. We validate NPO through two manual interventions, early-stage bootstrapping and late-stage plateau breakthrough, and further propose \textbf{AutoNPO},an adaptive variant that automatically triggers interventions from online training signals and selects the guide checkpoint that maximizes $S$. On Qwen3-VL-8B-Instruct with GRPO, NPO improves average performance from 57.88 to 62.84, and AutoNPO pushes it to 63.15, raising the final performance ceiling while accelerating convergence.
BibTeX
@misc{qin2026near,
  author = {Chuanyu Qin and Chenxu Yang and Qingyi Si and Naibin Gu and Dingyu Yao and Zheng Lin and Peng Fu and Nan Duan and Jiaqi Wang},
  title = {{Near-Future Policy Optimization}},
  year = {2026},
  eprint = {2604.20733},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.20733},
}

Loading related content...

Figure