A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning

Yuzheng Hu, Fan Wu, Haotian Ye, David Forsyth, James Zou, Nan Jiang, Jiaqi W. Ma, Han Zhao

TL;DR

Online reinforcement learning (RL) is notoriously unstable and sample-inefficient, largely because some collected experiences actually hinder the agent's learning. This paper introduces the first local data attribution framework for online RL, allowing researchers to trace policy changes back to specific experience records in the training buffer. By filtering out these harmful, negative-influence experiences during training, their proposed algorithm—Iterative Influence-Based Filtering (IIF)—significantly accelerates training and boosts final performance across standard RL benchmarks and LLM alignment (RLHF).

What Problem Does This Paper Address?

Unlike supervised learning, which relies on static datasets, online RL operates in a continuous "chicken-and-egg" loop: the agent's current policy collects data from the environment, and that data is immediately used to update the policy, which then dictates what data is collected next.

Because of this circular dependency, traditional data attribution methods (which trace model behavior back to specific training samples) fall flat. They assume a fixed dataset and objective, failing to account for how a single experience shapes all future data collection. As a result, online RL has remained a black box, making it incredibly difficult to diagnose why an agent suddenly destabilizes or learns inefficiently. The authors set out to answer a fundamental question: Which specific experiences in our recent rollout buffer are helping or hurting the policy update right now?

Methods

The authors introduce a local data attribution framework tailored for Proximal Policy Optimization (PPO). Instead of trying to trace influence across the entire, chaotic history of training, they isolate a single training round on a fixed rollout buffer.

Figure 1: An Illustration of the alternating learning cycle in online RL (Sec. 2.1) and our local data attribution framework (Sec. 3.1). Online RL operates in alternating rounds of data collection and policy updates; our local data attribution framework quantifies how individual records from a single round influence different aspects of policy update in that round.

They adapt TracIn—a gradient-based data attribution method—to calculate an "influence score" for individual experience records. This score measures the inner product (similarity) between the gradient of a record's training loss and the gradient of a chosen target function. They design two novel target functions:

  1. Agent Action ($f^{\text{action}}$): Used for diagnosis, tracing why an agent prefers a specific action in a given state.
  2. Cumulative Return ($f^{\text{return}}$): A stable surrogate objective based on REINFORCE with a baseline, using the policy at the start of the round as a reference to evaluate overall performance contribution without introducing massive variance.

Leveraging this framework, they propose Iterative Influence-Based Filtering (IIF). Between the data collection and policy update phases, IIF computes the influence of all records, identifies the bottom $p\%$ of negative-influence records, and discards them. To prevent this from blowing up computational costs, IIF computes influence using only the initial parameters of that round and utilizes an efficient "ghost dot product" trick.

Datasets

The framework was rigorously evaluated across a diverse suite of environments:

  • Navigation: FrozenLake, MiniGrid
  • Classic Control: Acrobot, LunarLander
  • Driving: Highway
  • Locomotion: BipedalWalker
  • LLM Alignment (RLHF): Detoxifying a 2.7B parameter GPT-Neo model using the real-toxicity-prompts dataset, with evaluation conducted on the distinct Wiki-Toxic dataset.

Key Insights and Ideas

  • Harmful Data = Bad Math: When analyzing the "bottom records" (the data most detrimental to learning), the authors uncovered a striking pattern: these records are plagued by inaccurate advantage estimates. They either flip the sign of the true advantage (e.g., rewarding a catastrophic move) or introduce massive magnitude errors, sending highly misleading gradient signals to the model.
Figure 3: (a-b) Examples of bottom records. (a) Bottom 100 records in FrozenLake at k = 5, aggregated over (s, a) for demonstration: arrow indicates action, green/red for positive/negative ˆA. (b) Selected records among bottom 20 in MiniGrid at k = 5: ▼–agent, ■–goal, gray area–the limited egocentric observation, yellow arrows–agent action in {turn left, turn right, forward}; all records shown are of positive ˆA. (c-d) These records are harmful due to their inaccurate advantage estimates. We sort records by decreasing influence (top on the left). (c) y axis is | ¯A −ˆA|; points with same/opposite signs for ˆA and ¯A colored green/red; top/bottom 20% region shaded green/red, and the intermediate in gray. (d) The product ¯A · ˆA versus record rank, showing a strong negative correlation.
  • The Three Phases of Behavior Formation: By tracking top-influence records across training rounds, they discovered a distinct three-stage phase transition in how agents learn a behavior:
  • Action-Advantage Association: The agent blindly pairs actions with rewards, largely ignoring the state context.
  • Semantic Clustering: The agent begins to generalize, grouping semantically similar states together to form nuanced representations.
  • Influence Saturation: As the policy converges, influence scores stabilize near zero and become dominated by random noise.
  • Less is More: Uniformly treating all collected experiences as equal is suboptimal. Selectively purging the rollout buffer acts as a "signal cleanser," allowing the policy to update using only high-fidelity, constructive data.

Key Results

  • Substantial Sample Efficiency Gains: Across all standard RL benchmarks, IIF achieved a 20% to 67% reduction in the number of training rounds required to match standard PPO performance.
  • Wall-Clock Speedups: Even with the overhead of calculating influence scores, IIF drastically reduced total runtime—achieving up to a ~71.5% runtime reduction at peak performance in MiniGrid—because training on a smaller, filtered batch significantly speeds up the optimization phase.
  • RLHF Efficiency: When applied to LLM detoxification, IIF achieved an ~4× reduction in total runtime while converging to a policy with higher rewards and lower toxicity than standard PPO.

Limitations and Future Work

  • Optimizer Mismatch: TracIn is theoretically formulated for standard Stochastic Gradient Descent (SGD). However, modern RL and LLM training predominantly use adaptive optimizers like Adam. While using SGD as a proxy for influence calculation proved highly effective here, developing attribution methods natively tailored for adaptive optimizers is an important next step.
  • Lack of Long-Term Counterfactuals: Because the framework is local (focusing on a single round), it cannot perfectly predict the long-term causal impact of removing a record on future data rollouts.
  • Algorithmic Expansion: The current framework is built around PPO. Extending local data attribution to off-policy algorithms (like DQN) or alternative LLM alignment frameworks (like GRPO) remains an open and promising direction.

Summary updated

Abstract

Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability. Data attribution provides a principled way to trace model behavior back to training samples, yet existing methods assume fixed datasets, which is violated in online RL where each experience both updates the policy and shapes future data collection. In this paper, we initiate the study of data attribution for online RL, focusing on the widely used Proximal Policy Optimization (PPO) algorithm. We start by establishing a \emph{local} attribution framework, interpreting model checkpoints with respect to the records in the recent training buffer. We design two target functions, capturing agent action and cumulative return respectively, and measure each record's contribution through gradient similarity between its training loss and these targets. We demonstrate the power of this framework through three concrete applications: diagnosis of learning, temporal analysis of behavior formation, and targeted intervention during training. Leveraging this framework, we further propose an algorithm, iterative influence-based filtering (IIF), for online RL training that iteratively performs experience filtering to refine policy updates. Across standard RL benchmarks (classic control, navigation, locomotion) to RLHF for large language models, IIF reduces sample complexity, speeds up training, and achieves higher returns. Together, these results open a new direction for making online RL more interpretable, efficient, and effective.
BibTeX
@inproceedings{hu2025snapshot,
  author = {Yuzheng Hu and Fan Wu and Haotian Ye and David Forsyth and James Zou and Nan Jiang and Jiaqi W. Ma and Han Zhao},
  title = {{A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=sYK4yPDuT1},
}
Figure