A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

Audrey Huang, Nan Jiang, Sham M. Kakade, Sadhika Malladi, Akshay Krishnamurthy, Dylan J. FosterarXiv 2026 · arXiv

In the world of Large Language Models (LLMs), "post-training"—the phase where we use Reinforcement Learning (RL) to refine a pretrained base model—is where the magic happens. It’s how we get models to follow instructions, reason through math problems, and avoid harmful outputs. Yet, despite its success, the theoretical "why" behind many post-training recipes remains a black box.

A new paper, "A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets," by researchers from Harvard, UIUC, UCSD, and Microsoft, provides a rigorous look under the hood. They identify a critical failure mode called "Coverage Collapse" and prove why a simple intervention—periodically resetting your reference policy—is the key to efficient learning.

TL;DR

The authors show that standard RL algorithms like REINFORCE can cause LLMs to "forget" how to generate optimal responses during training, leading to stalled optimization. They introduce Approximate Policy Mirror Descent (APMD), a framework that uses "reference resets" to preserve the base model's capabilities while reaching global optimality with significantly fewer steps.

What Problem Does This Paper Address?

The central goal of post-training is to leverage a highly capable base model. If a base model already has "coverage"—meaning it can generate the correct answer at least some of the time—RL should theoretically be able to amplify that signal quickly.

However, the authors find that standard RL objectives are non-concave and prone to Coverage Collapse. In this scenario, as the model tries to improve, it might actually drive the probability of optimal responses to near zero before it manages to find them again. This creates a "reward plateau" where the gradient signal vanishes, and the model requires an exponential number of training steps to recover, even if the base model was already quite good.

Figure 2: Coverage collapse stalls optimization with a fixed reference policy. We compare REINFORCE (Eq. (2)) and APMD using exact gradients on a modified instance from Theorem 2.1. APMD runs PPO-NoClip (Algorithm 1) with a reference reset every K = 100 inner steps. (a) In REINFORCE (black), coverage collapse (red) causes a prolonged reward plateau (yellow). The shaded regions are schematic. APMD (blue) converges faster as coverage keeps improving (smaller Cπt). (b,c) The REINFORCE trajectory plotted on the optimization landscape, with surface height Jη and color indicating reward (b) or response coverage (c), over the 2D parameter plane defined in Appendix C.1.

Methods

The authors propose a meta-algorithm called Approximate Policy Mirror Descent (APMD). The core innovation is the Reference Reset.

In standard RLHF, we often regularize the model to stay close to the original base model using a KL-divergence penalty. The authors argue this is a mistake:

  1. Weak Regularization: Allows the model to drift too far, leading to coverage collapse.
  2. Strong Regularization: Keeps the model safe but prevents it from ever reaching the optimal policy because it's "tethered" too tightly to the suboptimal base model.

APMD solves this by iteratively resetting the reference policy to the most recent iterate. By regularizing strongly to a moving target, the algorithm ensures that each sub-problem is easy to solve and the model never loses its "coverage" of optimal responses.

Figure 1: Comparison of regularization strategies. (a) Weak regularization to a fixed base leads to collapse. (b) Strong regularization to a fixed base stalls far from the star. (c) APMD uses reference resets to reach the star through successive local improvements.

The authors apply this framework to analyze several popular algorithms:

  • PPO-NoClip & PPO-Lite: Stylized versions of Proximal Policy Optimization.
  • KimiPO: The squared-loss approach used by the Kimi reasoning models.
  • Rebel-OnPolicy: A variant of the Rebel algorithm that uses paired responses to eliminate the need for a baseline.

Datasets

As a theoretical work, the paper does not rely on large-scale empirical datasets. Instead, it uses a theoretical testbed of softmax linear policies over frozen pretrained features. To demonstrate the "Coverage Collapse" phenomenon, they construct a specific "single-prompt bandit" instance where they can mathematically prove and visualize the failure of standard methods.

Key Insights and Ideas

  • The "Aha" Moment on Regularization: Regularization isn't just about preventing "reward hacking"; it's about controlling the optimization landscape. By resetting the reference, you keep the model in a "local neighborhood" where the RL objective behaves more like a nice, convex function.
  • Clipping is Statistical, Not Just Algorithmic: In PPO, "clipping" the importance weights is often seen as a trick to stabilize gradients. This paper suggests a different benefit: clipping limits the "statistical cost" of large importance weights, allowing for smaller batch sizes even when the policy shifts significantly.
  • Implicit vs. Explicit Regularization: Algorithms like KimiPO (which use squared loss) don't have an explicit KL penalty, but the authors prove they implement a form of implicit regularization to the reference policy, making them "coverage-preserving" by nature.

Key Results

  • Exponential vs. Polynomial: The authors prove that while REINFORCE can take exponentially many steps to converge (relative to the inverse of the target accuracy), APMD-based algorithms converge in polynomial time.
  • First Global Guarantees: This paper provides the first end-to-end global convergence guarantees for several widely used post-training families, including the methods used to train the Kimi models.
  • Coverage Preservation: They formally define "coverage-preserving" algorithms—those whose training cost is governed by how good the base model is. They show that reference resets are the necessary ingredient to achieve this.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Linear Simplification: The end-to-end guarantees are currently limited to softmax linear policies. Extending this to complex, non-linear neural networks is a major next step.
  • One-Shot Reasoning: The theory focuses on "one-shot" tasks (like a single bandit arm pull). Real-world LLM reasoning often involves multi-step interactions, which adds layers of complexity to the optimization landscape.
  • Discovery vs. Amplification: The paper focuses on amplifying capabilities already present in the base model. A separate, harder question is how RL helps a model discover entirely new behaviors that were never seen during pretraining.

By providing a theoretical bridge between classical optimization and modern LLM finetuning, this work offers a principled path forward for designing more stable and efficient post-training pipelines.

Summary updated

Abstract

Policy optimization is central to language model post-training, yet principled guidance remains limited. We revisit its theoretical foundations to identify algorithms that learn near-optimal policies using total samples and gradient steps governed by the base model's response coverage, a measure of how readily it generates optimal responses. We call such algorithms coverage-preserving. This guarantee captures the goal of amplifying capabilities accessible through sampling from the base model. (1) Coverage collapse. REINFORCE, the basis for PPO and GRPO, can drive the probability of optimal responses to near zero, even as reward improves. This coverage collapse stalls optimization, and convergence can require exponentially many steps despite good initial coverage. The failure arises even with exact gradients and persists under KL regularization to the base model. (2) Reference resets. We show that repeatedly resetting the KL reference to the most recent policy mitigates coverage collapse. We formalize this intervention in Approximate Policy Mirror Descent (APMD), a meta-algorithm that approximately solves a sequence of KL-regularized RL subproblems. APMD enables stronger regularization, making each subproblem easier to solve, while reference resets allow convergence to global optimality. (3) Coverage-preserving algorithms. For softmax linear policies over pretrained features, we use APMD to establish the first global convergence guarantees showing that widely used families of post-training algorithms are coverage-preserving. These include stylized PPO and GRPO variants with explicit KL regularization and reference resets, and squared-loss algorithms that implement them implicitly, including those used to train the Kimi models. These results clarify how regularization, clipping, and objective choice affect the sampling and optimization costs of post-training.
BibTeX
@misc{huang2026theory,
  author = {Audrey Huang and Nan Jiang and Sham M. Kakade and Sadhika Malladi and Akshay Krishnamurthy and Dylan J. Foster},
  title = {{A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets}},
  year = {2026},
  eprint = {2611.14467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14467},
}
Figure