A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points

Z. Q. Liu, Jie Lü, Guangquan Zhang, Junyu Xuan, Wenpeng Lu

Abstract

Deep Reinforcement Learning (DRL) typically assumes stationary environments where transition dynamics and reward functions remain constant. However, real-world applications often involve non-stationary conditions, such as changing terrain for robots or shifting market dynamics for trading agents. We present BADA (Behavior-Aware Detection and Adaptation), a framework designed to detect environmental changes and adapt policies without prior knowledge of change points or manual sensitivity thresholds. BADA monitors shifts in an agent's global behavior distribution using Wasserstein distance and statistical permutation tests. Once a change is detected, a contrastive adaptation mechanism steers the agent away from obsolete strategies. Evaluated on diverse ViZDoom scenarios, BADA demonstrates superior detection accuracy and faster recovery compared to state-of-the-art baselines.

Introduction: The Challenge of Non-stationarity

Most DRL algorithms are built on the assumption that the "rules of the game" are fixed. In reality, environments are frequently non-stationary. A robot might transition from a smooth floor to a slippery one, or an autonomous vehicle might encounter varying weather conditions. Existing solutions often require "change points" to be known in advance or rely on noisy signals like reward drops, which often necessitate brittle, manually-tuned thresholds. These methods frequently suffer from catastrophic forgetting or remain trapped in suboptimal behaviors learned in previous environments.

The BADA Framework

BADA treats an agent's holistic behavior as a sensor for environmental change. Instead of analyzing local state-action pairs, it considers the distribution of entire trajectories over time.

Behavior Embedding and Detection

BADA maps sequences of states, actions, and rewards into a latent "behavior space" using a neural network. To identify environmental shifts, the framework calculates the Wasserstein distance between behavior distributions of consecutive training epochs.

Conceptual overview of BADA, mapping agent trajectories into a behavior embedding space to detect environmental shifts.

To eliminate the need for manual thresholds, BADA employs a statistical permutation test. By shuffling behavior data and checking for statistical significance (low p-values), the system automatically confirms whether a shift in behavior reflects a true environmental change.

Contrastive Adaptation

Upon detecting a change, BADA introduces an adaptation regularization term to the objective function. This term explicitly pushes the agent's policy away from its previous optimal behavior. This "unlearning" process prevents the agent from hesitating near old, now-useless strategies, facilitating faster exploration and mastery of the new environment.

The BADA workflow, illustrating the detection of distribution shifts and the subsequent adaptation using regularization and trust region constraints.

Experimental Evaluation

The framework was tested using the ViZDoom platform across four non-stationary scenarios, including sudden changes in lighting, enemy counts, and map geometry. These scenarios test the agent's ability to recognize when the environment has shifted and how quickly it can master the new task.

Key Results and Insights

BADA outperformed baselines in both detection accuracy and recovery speed. In the "Basic" scenario, BADA achieved an F1 score of 0.95, significantly higher than the 0.80 achieved by the next best unsupervised method. Furthermore, BADA maintained high average rewards even when environmental changes occurred frequently (up to 9 times per run), whereas competing methods often saw performance collapse.

The success of BADA highlights two critical insights:

  1. Behavior as a Global Signal: Trajectory distributions are more reliable indicators of change than individual state-action pairs. Two environments might look identical at a single moment, but the distribution of trajectories will reveal the shift.
  2. Threshold-Free Robustness: Permutation tests allow the system to adapt its sensitivity automatically, making it more generalizable across different tasks without human intervention.

Conclusion

BADA provides a robust, threshold-free solution for DRL in unpredictable environments. By leveraging behavior embeddings and statistical tests, it bridges the gap between stationary training and the dynamic reality of the real world. Future work will explore off-policy adaptation to further enhance sample efficiency.

Summary updated

BibTeX
@inproceedings{liu2024behavior,
  author = {Z. Q. Liu and Jie Lü and Guangquan Zhang and Junyu Xuan and Wenpeng Lu},
  title = {{A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=Wrd2dKvP0Q},
}
Figure