Recycling Experience for Efficient Policy Optimization

This direction is split into other directions: on-policy sample inefficiency (on-policy data inefficiency). Its last version stays readable and citable here.

Version 1 of 1231 papers236 conceptsfrozen 2026-10-04

In the standard workflow of Reinforcement Learning (RL), an agent interacts with an environment, collects a batch of trajectories, updates its policy, and then immediately throws that data away. This "on-policy" constraint, popularized by algorithms like Proximal Policy Optimization (PPO) by John Schulman and colleagues, ensures stability by only learning from data that reflects the agent's current behavior.

However, this stability comes at a steep price: sample inefficiency. For modern applications like training Large Language Models (LLMs) to reason, generating a single trajectory can be computationally expensive. Discarding these "soaked sponges" after one squeeze is a luxury the field can no longer afford. A growing body of research focuses on breaking this on-policy shackle, allowing models to "recycle" past experiences to accelerate learning without collapsing into instability.

The On-Policy Bottleneck

The fundamental problem with on-policy learning is the distribution shift. If a policy $\pi$ changes even slightly, the probability of taking certain actions changes. Data collected by an older version of the policy ($\pi_{old}$) is no longer a perfect representative of what the current policy would do. If you try to update the current policy using that old data without correction, the gradients become biased, often leading to catastrophic forgetting or policy collapse.

As Takeshi Mori and Yutaka Nakamura noted in Off-Policy Natural Actor-Critic, this forces agents to spend the vast majority of their time collecting new data rather than actually learning from it. In complex domains like humanoid locomotion, Xue Bin Peng highlights in MimicKit that even with GPU acceleration, this data requirement remains a primary bottleneck.

Bridging the Gap with Off-Policy Critics

One way to solve this is to use an "off-policy critic"—a secondary model that estimates the value of states and actions regardless of which policy generated them. Shixiang Shane Gu and Timothy P. Lillicrap pioneered this with Q-Prop.

Q-Prop uses a Taylor expansion of an off-policy critic as a "control variate." Think of this as a baseline that "subtracts out" the high variance of the policy gradient. Because the critic can learn from any data in a replay buffer, it provides a stable signal that helps the on-policy actor learn much faster. This approach was further refined in Soft Actor-Critic (SAC) by Tuomas Haarnoja and Sergey Levine, which maximizes both reward and entropy to ensure the agent explores broadly while reusing every transition stored in its memory.

Safe Reuse via Trust Regions

To reuse data safely, researchers have developed "off-policy" versions of trust-region methods. The goal is to ensure that the update from old data doesn't push the policy too far from where it was.

In Off-Policy Proximal Policy Optimization, Wenjia Meng and Gang Pan introduced a way to scale PPO's famous clipping bounds based on the "behavior policy" (the policy that originally collected the data). This allows the agent to learn from a replay buffer while maintaining the rock-solid stability guarantees of the original PPO. Similarly, James Queeney and Ioannis Paschalidis proposed Generalized Policy Improvement (GPI), which defines a "generalized trust region" that theoretically supports sample reuse across multiple past policies.

Squeezing the Sponge for LLMs

The most recent frontier for these techniques is the training of Large Reasoning Models (LRMs). When an LLM generates a long chain-of-thought to solve a math problem, that trajectory is a goldmine of information.

Hongyao Tang and Jinyi Liu introduced ReMix (Reincarnating Mix-policy PPO) in Squeeze the Soaked Sponge, demonstrating that by mixing current and historical trajectories, they could achieve state-of-the-art reasoning performance with 30x to 450x less rollout data than on-policy methods.

This efficiency is often achieved by being selective about which experiences to recycle. Runzhe Zhan's ExGRPO prioritizes "medium-difficulty" questions and "low-entropy" (high-confidence) reasoning paths from the past. By focusing on the most informative historical samples, the model avoids the "stalling" problem identified in COPO by Jinghang Han, where models stop learning if all sampled responses for a prompt are identical.

Why It Works: The Intuition of Baselines

The core intuition behind these methods is that information is not the same as interaction. An interaction (a rollout) is expensive, but the information within it—the relationship between a state, an action, and a reward—is a persistent truth about the environment.

By using techniques like importance sampling or off-policy critics, researchers are essentially building better "baselines." If we know what a "typical" successful trajectory looks like from our history, we can more accurately judge whether our current policy's slight variation is actually an improvement. As Charles Arnal and Remi Munos show in Asymmetric REINFORCE, this allows us to "conservatively" learn from successes in the past while avoiding the catastrophic collapse that comes from trying to penalize failures that our current policy might not even make anymore.

Papers in this version

Showing 60 of 231 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recycling_experience_for_efficient_polic_98f1bc17,
  title = {Recycling Experience for Efficient Policy Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recycling-experience-for-efficient-policy-optimization/rdr_98f1bc17f62747619063101a9d3e1dc1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_recycling_experience_for_efficient_polic_98f1bc17_v1,
  title = {Recycling Experience for Efficient Policy Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recycling-experience-for-efficient-policy-optimization/rdr_98f1bc17f62747619063101a9d3e1dc1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}