Primacy Bias and the Loss of Plasticity

31 concepts25 papers1 proposal

In reinforcement learning (RL), an agent acts as both the student and the data collector. This dual role creates a fundamental bootstrapping problem. At the start of training, the agent's policy is highly suboptimal, meaning the initial experiences it records are noisy, low-quality, and unrepresentative of the true task dynamics. Yet, standard deep learning frameworks immediately begin optimizing the network's weights to fit this early data.

As identified by Evgenii Nikishin et al. in The Primacy Bias in Deep Reinforcement Learning, this dynamic triggers a phenomenon known as primacy bias: the agent overfits to its earliest interactions, permanently stunting its ability to incorporate better data discovered later in training. Instead of continuously improving, the agent becomes trapped in a performance plateau, executing rigid, suboptimal strategies despite continued interaction with the environment.

Why Networks Lose Their Plasticity

Why can't a powerful neural network simply overwrite these early, poor strategies when it eventually stumbles upon better ones? The answer lies in a progressive rigidity in the network's internal representations, often termed a loss of plasticity.

When a network overfits to early data, its gradient landscape changes drastically. Xingwu Chen et al. demonstrate in Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection that a strong early bias toward a suboptimal pattern leads to an "entanglement phase." During this phase, gradient updates are dominated by the existing high-probability, low-accuracy paths, making it mathematically difficult for the model to transition to superior strategies.

This entrenchment happens at both the structural and neuronal levels:

The Replay Buffer Trap

One might assume that the experience replay buffer—a core component of off-policy RL designed to break temporal correlations—would alleviate this issue. In reality, it often acts as an anchor that compounds primacy bias.

If an agent greedily stores and replays its early successful trajectories, it introduces a severe bias. As Daochen Zha et al. observe in Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments, the agent can get stuck imitating early strategies that are no longer optimal as the policy evolves.

Crucially, Zilin Kang et al. reveal in A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control that simply resetting the network's weights is insufficient to cure primacy bias. Because the replay buffer remains saturated with biased, low-quality early data, a reset network will quickly re-learn the exact same suboptimal patterns. True mitigation requires addressing both the network's weights and the temporal distribution of the memory buffer.

Mitigation Strategies

To overcome primacy bias, researchers have developed mechanisms to restore or maintain network plasticity, broadly falling into three categories:

1. Periodic Parameter Resets

The most direct way to break representation entrenchment is to clear the network's memory. Evgenii Nikishin et al. show that periodically re-initializing the final layers of a network while keeping the replay buffer intact restores the agent's ability to learn.

This concept extends beyond standard RL. In large language model fine-tuning, Zichuan Liu demonstrates in Sample-efficient LLM Optimization with Reset Replay that periodic parameter resets prevent the permanent degradation of reasoning quality caused by over-training on initial batches. Similarly, Chenyang Wu et al. use periodic resets in Efficient and Stable Offline-to-online Reinforcement Learning via Continual Policy Revitalization to counteract the brittleness and loss of plasticity induced by static offline pre-training.

2. Co-managing Weights and Memory

Because the replay buffer anchors bad habits, effective scaling requires a dual approach. Zilin Kang et al. propose the "Forget-and-Grow" (FoG) strategy, which actively manages both the neural network's weights and the temporal distribution of experiences in the memory buffer to mimic biological forgetting.

In a similar vein, Gong Xudong et al. argue in Iterative Regularized Policy Optimization with Imperfect Demonstrations that retraining a model entirely from scratch on high-quality generated data is far more effective than continuing to fine-tune existing weights, as it completely bypasses the sub-optimal habits learned early on.

3. Continuous Plasticity Preservation

Rather than relying on disruptive hard resets, some approaches maintain plasticity continuously through architectural constraints. Daniel Palenicek et al. demonstrate in Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization that combining batch and weight normalization stabilizes the effective learning rate. This prevents the network from losing its plasticity in the first place, allowing agents to handle high update-to-data ratios without needing large ensembles or resets.

Alternatively, Md. Masudur Rahman 0001 and Yexiang Xue show in Robust Policy Optimization in Deep Reinforcement Learning that adding a small uniform random perturbation to the action distribution prevents early entropy collapse, allowing agents to overcome primacy bias and sustain learning over high-throughput data streams.

Key Foundational Work

The Primacy Bias in Deep Reinforcement Learning (2022) — contributed 2 concept elements to this direction.
Robust Policy Optimization in Deep Reinforcement Learning (2022) — contributed 2 concept elements to this direction.
Dynamic Sparse Training for Deep Reinforcement Learning (2022) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 26 papers
  2. Version 1 · 2026-10-04 · 51 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_primacy_bias_acb09902,
  title = {primacy bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/primacy-bias/rdr_acb099025c8945fba0264fd5f96ca50b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_primacy_bias_acb09902_v2,
  title = {primacy bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/primacy-bias/rdr_acb099025c8945fba0264fd5f96ca50b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake