A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points

Zihe Liu, Jie Lu, Guangquan Zhang, Junyu Xuan

Abstract

Deep reinforcement learning is used in various domains, but usually under the assumption that the environment has stationary conditions like transitions and state distributions. When this assumption is not met, performance suffers. For this reason, tracking continuous environmental changes and adapting to unpredictable conditions is challenging yet crucial because it ensures that systems remain reliable and flexible in practical scenarios. Our research introduces Behavior-Aware Detection and Adaptation (BADA), an innovative framework that merges environmental change detection with behavior adaptation. The key inspiration behind our method is that policies exhibit different global behaviors in changing environments. Specifically, environmental changes are identified by analyzing variations between behaviors using Wasserstein distances without manually set thresholds. The model adapts to the new environment through behavior regularization based on the extent of changes. The results of a series of experiments demonstrate better performance relative to several current algorithms. This research also indicates significant potential for tackling this long-standing challenge.

Introduction

Standard Deep Reinforcement Learning (RL) algorithms typically operate under the assumption of stationarity, where environmental dynamics and reward structures remain constant. However, real-world applications—ranging from robotics to financial trading—frequently encounter non-stationary conditions. When environmental shifts occur, agents trained on previous conditions often experience significant performance degradation.

The challenge in non-stationary RL is twofold: detecting that a change has occurred without explicit signals (change points) and rapidly adapting the policy to the new environment. Existing methods often rely on noisy signals like reward drops or require manually tuned thresholds that fail to generalize. We propose Behavior-Aware Detection and Adaptation (BADA), which leverages the agent's global behavior as a reliable sensor for environmental shifts.

Conceptual overview of BADA mapping trajectories to a behavior embedding space to detect environmental shifts.

Methods

BADA consists of two primary components: behavior-based change detection and behavior-aware adaptation.

Behavior-Based Change Detection

Instead of monitoring local state transitions, BADA maps entire trajectories (sequences of states, actions, and rewards) into a latent behavior embedding space using a neural network. To detect shifts, we employ:

  • Wasserstein Distance: This metric quantifies the difference between behavior distributions across training epochs. Unlike KL divergence, it provides a meaningful gradient even for non-overlapping distributions.
  • Permutation Test: To eliminate the need for manual thresholds, BADA uses a statistical permutation test to calculate a p-value. This ensures that only statistically significant behavioral changes are flagged as environmental shifts.

Behavior-Aware Adaptation

Once a change is detected, BADA modifies the RL objective function to facilitate rapid pivoting. The adaptation mechanism uses a two-part regularization strategy:

  1. Trust Region Constraint: Maintains stability by keeping the new policy close to the immediate previous epoch.
  2. Adaptation Regularization: A contrastive term that actively pushes the policy away from the behavior that was optimal in the previous environment. The strength of this push is scaled by a coefficient derived from the magnitude of the detected environmental shift.
The BADA framework architecture illustrating the detection of shifts and the subsequent adaptation via trust region and regularization constraints.

Experimental Evaluation

We evaluated BADA using the ViZDoom platform across four non-stationary scenarios: basic (lighting/texture changes), defend_the_line (geometry changes), deadly_corridor (enemy count changes), and health_gathering (visual kit changes).

Key Results

  • Detection Accuracy: BADA achieved superior F1 scores for change detection, reaching 0.95 in the basic scenario, significantly outperforming baselines like CRL-Unsup (0.80).
  • Adaptation Speed: Reward curves demonstrated that BADA recovers performance much faster than standard algorithms like PPO or TRPO following a change point.
  • Robustness: BADA maintained high average rewards even as the frequency of environmental changes increased, whereas baseline performance typically collapsed under high-frequency shifts.

Conclusion

BADA introduces a robust, threshold-free approach to non-stationary RL by treating an agent's global behavior as an indicator of environmental change. By combining Wasserstein-based detection with active "unlearning" of obsolete strategies, BADA enables agents to thrive in unpredictable environments. Future work will explore off-policy mechanisms to further enhance sample efficiency during adaptation phases.

Summary updated

Abstract

Deep reinforcement learning is used in various domains, but usually under the assumption that the environment has stationary conditions like transitions and state distributions. When this assumption is not met, performance suffers. For this reason, tracking continuous environmental changes and adapting to unpredictable conditions is challenging yet crucial because it ensures that systems remain reliable and flexible in practical scenarios. Our research introduces Behavior-Aware Detection and Adaptation (BADA), an innovative framework that merges environmental change detection with behavior adaptation. The key inspiration behind our method is that policies exhibit different global behaviors in changing environments. Specifically, environmental changes are identified by analyzing variations between behaviors using Wasserstein distances without manually set thresholds. The model adapts to the new environment through behavior regularization based on the extent of changes. The results of a series of experiments demonstrate better performance relative to several current algorithms. This research also indicates significant potential for tackling this long-standing challenge.
BibTeX
@inproceedings{liu2024behavior,
  author = {Zihe Liu and Jie Lu and Guangquan Zhang and Junyu Xuan},
  title = {{A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=Wrd2dKvP0Q},
}
Figure