Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Zhiwei Wang, Yanxi Chen, Yaliang Li, Bolin DingarXiv 2026 · arXiv

TL;DR

This paper provides a theoretical foundation for reward-guided self-training (like ReST or RAFT used in LLMs) by analyzing RE(S), a generalized REINFORCE algorithm that updates its rollout distribution every $S$ steps. The authors prove that not only does this "off-policy" approach converge globally, but it can also significantly outperform standard on-policy methods by helping models escape suboptimal "traps" during training.

What Problem Does This Paper Address?

In the world of Reinforcement Learning (RL), "on-policy" sampling is often treated as the gold standard. The idea is simple: to improve a policy, you should collect data using your current best version of that policy. If your data is "stale" (collected by an older version of the model), theory suggests you might need complex corrections like importance sampling to avoid divergence.

However, in modern Large Language Model (LLM) post-training, researchers often use "stale" data. Algorithms like STaR, RAFT, and ReST generate a large batch of data, then fine-tune the model on that fixed batch for several epochs before generating new data. This is computationally efficient, but does it actually work from a theoretical standpoint?

This paper bridges the gap between this common industrial practice and RL theory, asking: How does rollout staleness affect learning dynamics, and can "off-policyness" actually be a feature rather than a bug?

Methods

The authors introduce and analyze RE(S), a generalized version of the classic REINFORCE algorithm.

  • The "S" Parameter: In RE(S), the model performs $S$ gradient updates on a fixed set of rollout data before refreshing that data. When $S=1$, it reduces to standard on-policy REINFORCE.
  • Stage-wise Optimization: The authors show that RE(S) can be viewed as a stage-wise process. In each stage, the model takes $S$ steps to minimize the Kullback–Leibler (KL) distance between its current policy and a "reward-weighted" version of the rollout policy.
  • Theoretical Setting: The analysis focuses on $K$-arm bandits with softmax policies. While simpler than full MDPs, this setting allows for rigorous proofs regarding global convergence and specific convergence rates.

Datasets

As a theoretical work, the paper does not use standard large-scale datasets like ImageNet or Pile. Instead, it utilizes synthetic multi-arm bandit environments to validate its theorems. These environments are carefully constructed with specific reward vectors (e.g., $K=100$ arms with one optimal action) and varying initial policy strengths to test how the algorithm handles "traps" in the optimization landscape.

Key Insights and Ideas

  • Global Convergence is Robust: The paper proves that for any fixed $S$, RE(S) will eventually converge to the optimal policy. You don't need to update your rollout distribution every single step to guarantee success.
  • The "Burn-in" vs. Asymptotic Phase: The total number of gradient steps $T$ is what ultimately dictates the convergence rate, which is $\Theta(1/T)$. The staleness parameter $S$ primarily affects the length of the initial "burn-in" phase. Once past that, all versions of RE(S) converge at roughly the same rate.
Empirical validation of RE(S) convergence rates. As $T$ grows, different $S$ values converge to the same $1/T$ slope.
  • Escaping the Softmax Trap: This is the most striking finding. Softmax policies often get "stuck" near suboptimal actions because the gradient for the optimal action becomes vanishingly small (the "plateau" problem).
  • On-policy RE(1) gets trapped because it keeps sampling from the bad policy, reinforcing the plateau.
  • Off-policy RE(S), by keeping the rollout distribution fixed for a while, maintains a stronger "signal" for the optimal action, allowing the model to "tunnel" through the plateau much faster.

Key Results

  • Asymptotic Rate: The suboptimality gap achieves a $\Theta(1/T)$ rate regardless of $S$.
  • Provable Speedup: For a weak initial policy with optimal-action probability $x$, the authors prove that on-policy RE(1) can take $\Omega(x^{-(K-1)})$ steps to escape a trap. In contrast, RE(S) with a suitable $S$ can reach the optimum in just $O(x^{-1} \log(1/x))$ steps.
  • Numerical Validation: Experiments show that while RE(1) plateaus for a long period, RE(512) makes steady progress toward the global optimum.
Left: RE(1) plateaus at a suboptimal gap while RE(512) converges. Right: Visualization of the "detour" taken by on-policy methods in a 3-action simplex.

Limitations and Future Work

  • Bandit Setting: The current theory is limited to multi-arm bandits. Extending this to Markov Decision Processes (MDPs) where state transitions matter is a crucial next step.
  • Softmax Specifics: The benefits of off-policyness are closely tied to the geometry of softmax policies. It remains to be seen if similar benefits exist for other parameterizations.
  • Infinite Samples: The analysis assumes an infinite-sample limit for gradients. Future work needs to tackle the "sample complexity"—how many actual samples $N$ are needed to maintain these benefits in noisy, real-world settings.
  • LLM Realism: While the work is inspired by LLM fine-tuning, the gap between a $K$-arm bandit and a 70B parameter transformer is vast. Empirical verification in large-scale language tasks is still needed to confirm if these theoretical "traps" are the primary bottleneck in practice.

Summary updated

Abstract

We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version of REINFORCE -- referred to as RE(S) -- that updates the rollout distribution once every $S \ge 1$ gradient steps. Prior work in bandits and reinforcement learning has developed rich theory for policy gradient methods, and on-policy sampling (i.e., a small $S$, ideally $1$) is often viewed as crucial to their success; yet in prominent application like post-training large language models, reward-guided self-training has proved to be effective even when the rollout distribution is updated infrequently, but theoretical understanding remains limited for the convergence properties of these off-policy methods. To bridge these gaps, we develop a unified theory for RE(S) that covers the full spectrum of $S \ge 1$: it can be interpreted as a stage-wise optimization process, where each stage takes $S$ gradient steps for minimizing the Kullback-Leibler distance to a fixed reward-weighted rollout distribution. For multi-arm bandits with softmax policies, our in-depth analysis and numerical experiments reveal three key findings: (1) for any fixed $S$, RE(S) enjoys global convergence to the optimal policy as the number of rollout distribution updates $B = \lfloor T / S \rfloor \rightarrow \infty$, where $T$ denotes the number of gradient steps; (2) we prove tight two-sided bounds showing that the suboptimality gap of RE(S) achieves an asymptotic $Θ(1 / T)$ convergence rate, while $S$ only affects the length of a burn-in phase; (3) when initialized at a weak policy with a small optimal-action probability, RE(1) gets trapped around suboptimal policies for a long period, whereas RE(S) with a suitable $S$ avoids the detour and achieves significantly faster convergence to the global optimum, highlighting the benefits of off-policyness in this case.
BibTeX
@misc{wang2026fine,
  author = {Zhiwei Wang and Yanxi Chen and Yaliang Li and Bolin Ding},
  title = {{Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness}},
  year = {2026},
  eprint = {2609.36945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.36945},
}
Figure