Mitigating Non-Stationarity and Moving Targets in Reinforcement Learning

This direction is split into other directions: non-stationary data distribution. Its last version stays readable and citable here.

Version 1 of 1462 papers508 conceptsfrozen 2026-10-04

In supervised learning, we typically assume a static landscape: the dataset is fixed, and the labels are "ground truth." If a model sees a picture of a cat, the label "cat" doesn't change just because the model got better at recognizing dogs.

Reinforcement Learning (RL) breaks this assumption entirely. In RL, the agent is both the student and the architect of its own curriculum. As the agent learns, its behavior changes; as its behavior changes, the distribution of data it encounters shifts. This creates a fundamental challenge known as non-stationarity. This article explores the mechanisms behind this "moving target" problem and the strategies researchers use to stabilize it.

The Moving Target Problem

The most intuitive way to understand non-stationarity in RL is through the relationship between a policy (the "actor") and a value function (the "critic"). In the Actor-Critic framework, the critic tries to estimate how good the actor's current strategy is. However, as Zhengyuan Zhou and colleagues point out in Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic, the critic is chasing a moving target. By the time the critic has updated its estimates to reflect the actor's behavior, the actor has already moved on to a new, hopefully better, policy.

This co-dependency creates a feedback loop. If the critic’s estimates are slightly wrong, the actor might adopt a poor policy. This poor policy then generates a new distribution of data that reinforces the critic’s errors. As Navdeep Kumar explains in On the Convergence of Single-Timescale Actor-Critic, this interdependence is the primary reason why RL algorithms are notoriously difficult to converge compared to their supervised counterparts.

The Feedback Loop of Data Distribution

Beyond the internal math of value functions, there is a physical non-stationarity in the data itself. In the seminal Playing Atari with Deep Reinforcement Learning, the authors noted that RL violates the core assumption of Independent and Identically Distributed (IID) data.

Because an agent’s actions determine its future states, consecutive samples are highly correlated. If an agent is stuck in a corner of a maze, every frame it sees will look nearly identical. Furthermore, as the agent improves, it spends more time in "high-reward" states and less time in "low-reward" states. This shift means the neural network is constantly being asked to solve a different optimization problem. Emma L. Roscow notes in Learning offline that this makes it difficult for agents to maintain stable performance; they often "forget" how to handle states they haven't seen in a while.

The Cost of Adaptation: Plasticity Loss

When a neural network is forced to constantly adapt to shifting targets, it can lose its "plasticity"—the ability to learn new information. Clare Lyle and the team behind Understanding Plasticity in Neural Networks found that non-stationary environments cause the loss landscape to sharpen, making it harder for optimizers to find new optima.

A physical manifestation of this is the Dormant Neuron Phenomenon. As explored by Ghada Sokar in The Dormant Neuron Phenomenon in Deep Reinforcement Learning, neurons that were useful for an early version of the policy may become inactive as the policy evolves. These "dead" neurons stop contributing to the gradient, effectively shrinking the capacity of the network until it can no longer learn at all.

Strategies for Stabilization

To combat these issues, the field has developed several key "stabilizers" that are now standard in modern RL:

1. Breaking Correlations with Experience Replay

The most famous solution to data non-stationarity is the Experience Replay Buffer. By storing past transitions and sampling them randomly, agents can break the temporal correlation of their experiences. This effectively turns the non-stationary stream of data into a more "stationary-looking" dataset, allowing the network to learn from a mixture of old and new behaviors.

2. Slowing Down the Target

To solve the "critic chasing the actor" problem, researchers often use Target Networks. Instead of updating the value function's target at every step, they use a frozen "target" network that only updates periodically. This provides a stable objective for the main network to aim for. Recent work like StaQ it! by Alena Shilova extends this by using a "stack" of past Q-functions to achieve even smoother learning curves.

3. Normalization and Curvature Control

Because the loss landscape in RL is so volatile, standard optimizers like Adam can become brittle. Peter Henderson showed in Where Did My Optimum Go? that RL is uniquely sensitive to hyperparameters. To fix this, techniques like Spectral Normalization are used. Florin Gogianu demonstrates in Spectral Normalisation for Deep Reinforcement Learning that constraining the Lipschitz constant of the network prevents the gradients from exploding or vanishing as the target shifts.

4. Hierarchical Games

Some researchers treat the non-stationarity as a feature rather than a bug. The SCORER framework by Fernando J. Martínez in Stackelberg Coupling of Online Representation Learning and Reinforcement Learning reframes the relationship between the encoder and the value function as a Stackelberg game. By designating one component as a "slow leader" and the other as a "fast follower," they mathematically enforce a hierarchy that prevents the two from oscillating in an unstable loop.

Conclusion

Non-stationarity is the defining characteristic that separates Deep RL from other forms of machine learning. Whether it is the shifting data distribution or the co-dependent nature of value estimation, the "moving target" problem requires specialized architectures and algorithmic "tricks" to solve. By understanding these dynamics, researchers are moving away from heuristic engineering and toward a more principled theory of how agents can learn stably in an ever-changing world.

Papers in this version

Showing 60 of 462 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mitigating_non_stationarity_and_moving_t_6e75527b,
  title = {Mitigating Non-Stationarity and Moving Targets in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-non-stationarity-and-moving-targets-in-reinforcement-learning/rdr_6e75527b1aba467a99f237f32d40db1c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_non_stationarity_and_moving_t_6e75527b_v1,
  title = {Mitigating Non-Stationarity and Moving Targets in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-non-stationarity-and-moving-targets-in-reinforcement-learning/rdr_6e75527b1aba467a99f237f32d40db1c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}