In deep reinforcement learning (RL), an agent’s first few thousand interactions are its most dangerous. Because RL agents start as "blank slates," their initial experiences are dominated by random, low-quality exploration. Yet, due to the nature of gradient descent and iterative bootstrapping, these clumsy first steps can exert a disproportionate influence on the agent’s entire future.
This phenomenon is known as primacy bias: a tendency for neural networks to overfit to early data, creating rigid internal representations that resist new, higher-quality information discovered later in training.
The Trap of Early Success
At the start of training, an agent’s replay buffer is filled with "noisy" trajectories. If an agent happens to find a mediocre strategy that yields a small reward, the optimization process aggressively reinforces it. Max W. Shen and colleagues describe this as a reinforcement bias where the first discovered path is strengthened regardless of its long-term optimality.
This creates a self-reinforcing cycle. As Evgenii Nikishin, Aaron Courville, and their co-authors argue in The Primacy Bias in Deep Reinforcement Learning, this early overfitting stunts the agent's ability to incorporate better data. The network becomes "brittle," losing its plasticity—the fundamental ability to adapt its weights to new patterns.
The problem isn't the data itself; the experiences in the replay buffer are often sufficient to train a successful policy from scratch. Instead, the bottleneck is the optimization state. The network's weights become "entangled" in suboptimal patterns, making it mathematically difficult for subsequent gradients to shift the model toward a better global optimum.
Why Plasticity Vanishes
Why do neural networks stop being "teachable"? Several mechanisms contribute to this loss of plasticity:
- Feature Saturation: In Stay Hungry, Keep Learning, researchers note that neurons can become saturated by early experiences, making them unresponsive to the gradients produced by later, more relevant data.
- Spectral Collapse: Alex Lewandowski and Michal Bortkiewicz suggest that the singular values of the weight matrices can evolve in ways that reduce "gradient diversity," effectively locking the network into its current state.
- The Aleatoric Impulse: Zhouyang Yu and Shengbo Eben Li identify a transient surge in value overestimation during early training. Because RL relies on bootstrapping, this early "impulse" of error propagates through the value function, permanently derailing the agent's learning trajectory.
Strategies for Forgetting
To combat primacy bias, researchers have turned to a counter-intuitive solution: strategic forgetting. If the problem is that the network remembers its early mistakes too well, the solution is to force it to let go.
Periodic Resets
The most direct intervention is the "reset." Nikishin et al. demonstrated that periodically re-initializing the final layers of an agent’s network—while keeping its replay buffer intact—can break the cycle of overfitting and allow the agent to "re-learn" from its now-higher-quality data. To avoid the catastrophic performance drops associated with a full reset, Woojun Kim and Youngchul Sung proposed Reset Deep Ensembles (RDE), which uses a staggered ensemble of agents to maintain safety and performance while individual members are reset.
Biological Mimicry
Humans naturally experience "infantile amnesia," forgetting the specifics of early life to make room for more complex adult learning. Zilin Kang and Yu Luo translate this to RL with the Forget-and-Grow (FoG) strategy, which mimics brain development to help agents scale to complex tasks without getting stuck in the past. Similarly, Jiashun Liu introduces Neuroplastic Expansion, which starts with a small network and dynamically grows it, preventing early data from "filling up" the model's capacity too soon.
Information Scrubbing
Rather than resetting entire layers, Massimiliano Falzari and Matthia Sabatelli propose Fisher-Guided Selective Forgetting (FGSF). This method uses information geometry to identify specific weights that have become "stubbornly" biased toward early data and surgically scrubs them, preserving useful knowledge while restoring the network's ability to learn.
Beyond Robotics: LLMs and Diffusion
Primacy bias is not limited to traditional control tasks. In the world of Large Language Models (LLMs), Aviral Kumar and Vincent Zhuang found that standard RL fine-tuning often suffers from a "first-turn optimization bias," where models prioritize immediate rewards over learning long-term skills like self-correction. In generative modeling, Ziyi Zhang shows that diffusion models can "hack" reward functions due to stagnant internal representations in the critic model, a problem that can also be solved by periodically resetting active neurons.
By recognizing that "more training" is not always the answer, this research direction shifts the focus from data collection to plasticity management. For an agent to reach expert-level performance, it must not only learn from its experiences but also possess the structural flexibility to change its mind when it finds a better way.