TL;DR
Online reinforcement learning (RL) is notoriously unstable and sample-inefficient, largely because some collected experiences actually hinder the agent's learning. This paper introduces the first local data attribution framework for online RL, allowing researchers to trace policy changes back to specific experience records in the training buffer. By filtering out these harmful, negative-influence experiences during training, their proposed algorithm—Iterative Influence-Based Filtering (IIF)—significantly accelerates training and boosts final performance across standard RL benchmarks and LLM alignment (RLHF).
What Problem Does This Paper Address?
Unlike supervised learning, which relies on static datasets, online RL operates in a continuous "chicken-and-egg" loop: the agent's current policy collects data from the environment, and that data is immediately used to update the policy, which then dictates what data is collected next.
Because of this circular dependency, traditional data attribution methods (which trace model behavior back to specific training samples) fall flat. They assume a fixed dataset and objective, failing to account for how a single experience shapes all future data collection. As a result, online RL has remained a black box, making it incredibly difficult to diagnose why an agent suddenly destabilizes or learns inefficiently. The authors set out to answer a fundamental question: Which specific experiences in our recent rollout buffer are helping or hurting the policy update right now?
Methods
The authors introduce a local data attribution framework tailored for Proximal Policy Optimization (PPO). Instead of trying to trace influence across the entire, chaotic history of training, they isolate a single training round on a fixed rollout buffer.
They adapt TracIn—a gradient-based data attribution method—to calculate an "influence score" for individual experience records. This score measures the inner product (similarity) between the gradient of a record's training loss and the gradient of a chosen target function. They design two novel target functions:
- Agent Action ($f^{\text{action}}$): Used for diagnosis, tracing why an agent prefers a specific action in a given state.
- Cumulative Return ($f^{\text{return}}$): A stable surrogate objective based on REINFORCE with a baseline, using the policy at the start of the round as a reference to evaluate overall performance contribution without introducing massive variance.
Leveraging this framework, they propose Iterative Influence-Based Filtering (IIF). Between the data collection and policy update phases, IIF computes the influence of all records, identifies the bottom $p\%$ of negative-influence records, and discards them. To prevent this from blowing up computational costs, IIF computes influence using only the initial parameters of that round and utilizes an efficient "ghost dot product" trick.
Datasets
The framework was rigorously evaluated across a diverse suite of environments:
- Navigation: FrozenLake, MiniGrid
- Classic Control: Acrobot, LunarLander
- Driving: Highway
- Locomotion: BipedalWalker
- LLM Alignment (RLHF): Detoxifying a 2.7B parameter GPT-Neo model using the
real-toxicity-promptsdataset, with evaluation conducted on the distinctWiki-Toxicdataset.
Key Insights and Ideas
- Harmful Data = Bad Math: When analyzing the "bottom records" (the data most detrimental to learning), the authors uncovered a striking pattern: these records are plagued by inaccurate advantage estimates. They either flip the sign of the true advantage (e.g., rewarding a catastrophic move) or introduce massive magnitude errors, sending highly misleading gradient signals to the model.
- The Three Phases of Behavior Formation: By tracking top-influence records across training rounds, they discovered a distinct three-stage phase transition in how agents learn a behavior:
- Action-Advantage Association: The agent blindly pairs actions with rewards, largely ignoring the state context.
- Semantic Clustering: The agent begins to generalize, grouping semantically similar states together to form nuanced representations.
- Influence Saturation: As the policy converges, influence scores stabilize near zero and become dominated by random noise.
- Less is More: Uniformly treating all collected experiences as equal is suboptimal. Selectively purging the rollout buffer acts as a "signal cleanser," allowing the policy to update using only high-fidelity, constructive data.
Key Results
- Substantial Sample Efficiency Gains: Across all standard RL benchmarks, IIF achieved a 20% to 67% reduction in the number of training rounds required to match standard PPO performance.
- Wall-Clock Speedups: Even with the overhead of calculating influence scores, IIF drastically reduced total runtime—achieving up to a ~71.5% runtime reduction at peak performance in MiniGrid—because training on a smaller, filtered batch significantly speeds up the optimization phase.
- RLHF Efficiency: When applied to LLM detoxification, IIF achieved an ~4× reduction in total runtime while converging to a policy with higher rewards and lower toxicity than standard PPO.
Limitations and Future Work
- Optimizer Mismatch: TracIn is theoretically formulated for standard Stochastic Gradient Descent (SGD). However, modern RL and LLM training predominantly use adaptive optimizers like Adam. While using SGD as a proxy for influence calculation proved highly effective here, developing attribution methods natively tailored for adaptive optimizers is an important next step.
- Lack of Long-Term Counterfactuals: Because the framework is local (focusing on a single round), it cannot perfectly predict the long-term causal impact of removing a record on future data rollouts.
- Algorithmic Expansion: The current framework is built around PPO. Extending local data attribution to off-policy algorithms (like DQN) or alternative LLM alignment frameworks (like GRPO) remains an open and promising direction.
Summary updated