A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points

Zihe Liu, Jie Lu, Guangquan Zhang, Junyu Xuan

TL;DR

Deep reinforcement learning (DRL) agents typically struggle when deployed in real-world environments that change unpredictably, as most algorithms assume stationary conditions. This paper introduces Behavior-Aware Detection and Adaptation (BADA), a framework that automatically detects environmental shifts by measuring changes in an agent's global behavior using Wasserstein distance, without relying on manual thresholds. Once a change is detected, BADA actively forces the agent to deviate from its previous optimal behavior, leading to significantly faster adaptation and higher rewards in dynamic environments.

Introduction

Standard DRL algorithms are trained under the assumption that the underlying Markov Decision Process (MDP)—the state spaces, transition dynamics, and reward functions—remains fixed. However, real-world environments are inherently non-stationary. An outdoor robot might suddenly transition from flat pavement to rocky terrain, or a financial trading bot might face a sudden market crash.

Conceptual overview of BADA: Trajectories from different environments are mapped to distinct clusters in a behavior embedding space.

When environments change abruptly, standard RL policies often fail catastrophically. Existing solutions like continual learning or meta-RL usually require the "change points" (when the environment shifts) to be known in advance. Methods that attempt to actively detect these changes typically rely on simplistic metrics, like sudden drops in reward, which require brittle, manually tuned thresholds. Furthermore, relying purely on state changes is often insufficient, as different global behaviors might visit similar local states. This paper tackles the challenge of automatically detecting unknown environmental change points and rapidly adapting the policy without relying on hardcoded thresholds.

The BADA Framework

The authors propose BADA (Behavior-Aware Detection and Adaptation), an end-to-end framework that uses the agent's own behavior as a sensor for environmental change. The framework consists of three primary components:

  1. Behavior Embedding: The agent's trajectories (sequences of states, actions, and rewards) are mapped into a latent "behavior space" using a neural network. This allows the agent to represent complex temporal patterns as single points in a distribution.
  2. Threshold-Free Detection: To detect changes, BADA calculates the Wasserstein distance between the behavior distributions of adjacent training epochs. Because manually setting a threshold for this distance is difficult and environment-dependent, the authors employ a permutation test. This statistical test shuffles the data to calculate a p-value, allowing the system to infer if a true distribution shift occurred without needing a hardcoded threshold.
  3. Behavior-Aware Adaptation: Once a change point is detected, standard RL agents often hesitate to abandon their previously successful strategies, getting stuck in local optima. BADA solves this by adding a contrastive regularization term to the policy gradient objective. This term explicitly maximizes the difference between the agent's current behavior and its previously converged optimal behavior.
The BADA architecture showing the detection mechanism and the adaptation regularization used to force the agent away from outdated policies.

The weight of this penalty self-adjusts based on the Wasserstein distance of the detected change, ensuring the agent reacts aggressively to major shifts and gently to minor ones.

Experimental Evaluation

The authors evaluated BADA using ViZDoom, a 3D first-person shooter environment where agents learn using only visual inputs. They designed four custom non-stationary scenarios to simulate dynamic environments:

  • basic: The environment transitions from a high-contrast room to a dimly lit one.
  • defend_the_line/center: The map shape changes from rectangular to circular, and the defense goal shifts.
  • deadly_corridor: The number of enemies abruptly changes.
  • health_gathering: The textures of the health packs (medikits) are altered.

Key Results

  • Superior Detection Accuracy: BADA achieved the highest F1 scores for change detection across all scenarios. For example, in the basic lighting change scenario, BADA achieved an F1 score of 0.95, compared to 0.80 for the next best baseline (CRL-Unsup).
  • Faster Reward Recovery: Across all ViZDoom scenarios, BADA showed a significantly faster and steeper recovery in cumulative rewards post-change compared to baselines like PPO, TRPO, and BGPG.
  • Robustness to Frequent Changes: The authors tested the algorithms in environments with up to 9 sequential change points. While the performance of all methods degraded as the frequency of changes increased, BADA maintained a clear and significant lead in average reward over all competitors.

Discussion and Key Insights

  • Behavior is the Best Sensor: Global behavior trajectories are a much more robust and comprehensive indicator of environmental change than localized states or delayed reward signals.
  • Statistical Tests Beat Manual Thresholds: Using a permutation test on Wasserstein distances elegantly bypasses the tedious and brittle process of manually tuning detection thresholds for every new environment.
  • Forced Unlearning Accelerates Adaptation: By explicitly penalizing similarity to the old behavior once a change is detected, BADA forces the agent to rapidly explore new strategies rather than stubbornly clinging to a policy that no longer works.

Limitations and Future Work

BADA relies on comparing stable, converged behaviors. If an environment is extremely non-stationary (changing so frequently that the policy never converges), the agent's behavior remains chaotic, limiting detection accuracy. Future work may explore mechanisms for off-policy adaptations to broaden BADA's applicability to more sample-efficient RL settings.

Summary updated

BibTeX
@inproceedings{liu2024behavior,
  author = {Zihe Liu and Jie Lu and Guangquan Zhang and Junyu Xuan},
  title = {{A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=Wrd2dKvP0Q},
}
Figure