In reinforcement learning (RL), an agent acts as both the student and the data collector. This dual role creates a fundamental bootstrapping problem. At the start of training, the agent's policy is highly suboptimal, meaning the initial experiences it records are noisy, low-quality, and unrepresentative of the true task dynamics. Yet, standard deep learning frameworks immediately begin optimizing the network's weights to fit this early data.
As identified by Evgenii Nikishin et al. in The Primacy Bias in Deep Reinforcement Learning, this dynamic triggers a phenomenon known as primacy bias: the agent overfits to its earliest interactions, permanently stunting its ability to incorporate better data discovered later in training. Instead of continuously improving, the agent becomes trapped in a performance plateau, executing rigid, suboptimal strategies despite continued interaction with the environment.
Why Networks Lose Their Plasticity
Why can't a powerful neural network simply overwrite these early, poor strategies when it eventually stumbles upon better ones? The answer lies in a progressive rigidity in the network's internal representations, often termed a loss of plasticity.
When a network overfits to early data, its gradient landscape changes drastically. Xingwu Chen et al. demonstrate in Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection that a strong early bias toward a suboptimal pattern leads to an "entanglement phase." During this phase, gradient updates are dominated by the existing high-probability, low-accuracy paths, making it mathematically difficult for the model to transition to superior strategies.
This entrenchment happens at both the structural and neuronal levels:
- Structural Memorization: Ghada Sokar et al. note in Dynamic Sparse Training for Deep Reinforcement Learning that dense neural networks are highly susceptible to memorizing the noisy data collected during early exploration, locking the topology into suboptimal configurations.
- Neuronal Entrenchment: Ziyi Zhang et al. find in Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases that specific high-activity neurons become entrenched in early patterns, creating stagnant internal representations that resist new information.
- Internal Inertia: As Myungsik Cho explains in ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning, this over-reliance on initial network parameters creates an internal inertia that rejects new strategies even when task priorities or reward scales shift.
The Replay Buffer Trap
One might assume that the experience replay buffer—a core component of off-policy RL designed to break temporal correlations—would alleviate this issue. In reality, it often acts as an anchor that compounds primacy bias.
If an agent greedily stores and replays its early successful trajectories, it introduces a severe bias. As Daochen Zha et al. observe in Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments, the agent can get stuck imitating early strategies that are no longer optimal as the policy evolves.
Crucially, Zilin Kang et al. reveal in A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control that simply resetting the network's weights is insufficient to cure primacy bias. Because the replay buffer remains saturated with biased, low-quality early data, a reset network will quickly re-learn the exact same suboptimal patterns. True mitigation requires addressing both the network's weights and the temporal distribution of the memory buffer.
Mitigation Strategies
To overcome primacy bias, researchers have developed mechanisms to restore or maintain network plasticity, broadly falling into three categories:
1. Periodic Parameter Resets
The most direct way to break representation entrenchment is to clear the network's memory. Evgenii Nikishin et al. show that periodically re-initializing the final layers of a network while keeping the replay buffer intact restores the agent's ability to learn.
This concept extends beyond standard RL. In large language model fine-tuning, Zichuan Liu demonstrates in Sample-efficient LLM Optimization with Reset Replay that periodic parameter resets prevent the permanent degradation of reasoning quality caused by over-training on initial batches. Similarly, Chenyang Wu et al. use periodic resets in Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization to counteract the brittleness and loss of plasticity induced by static offline pre-training.
2. Co-managing Weights and Memory
Because the replay buffer anchors bad habits, effective scaling requires a dual approach. Zilin Kang et al. propose the "Forget-and-Grow" (FoG) strategy, which actively manages both the neural network's weights and the temporal distribution of experiences in the memory buffer to mimic biological forgetting.
In a similar vein, Gong Xudong et al. argue in Iterative Regularized Policy Optimization with Imperfect Demonstrations that retraining a model entirely from scratch on high-quality generated data is far more effective than continuing to fine-tune existing weights, as it completely bypasses the sub-optimal habits learned early on.
3. Continuous Plasticity Preservation
Rather than relying on disruptive hard resets, some approaches maintain plasticity continuously through architectural constraints. Daniel Palenicek et al. demonstrate in Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization that combining batch and weight normalization stabilizes the effective learning rate. This prevents the network from losing its plasticity in the first place, allowing agents to handle high update-to-data ratios without needing large ensembles or resets.
Alternatively, Md. Masudur Rahman 0001 and Yexiang Xue show in Robust Policy Optimization in Deep Reinforcement Learning that adding a small uniform random perturbation to the action distribution prevents early entropy collapse, allowing agents to overcome primacy bias and sustain learning over high-throughput data streams.
Key Foundational Work
The Primacy Bias in Deep Reinforcement Learning (2022) — contributed 2 concept elements to this direction.
Robust Policy Optimization in Deep Reinforcement Learning (2022) — contributed 2 concept elements to this direction.
Dynamic Sparse Training for Deep Reinforcement Learning (2022) — contributed 1 concept elements to this direction.