REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning

Philipp Altmann, Céline Davignon, Maximilian Zorn, Fabian Ritz, Claudia Linnhoff-Popien, Thomas Gabor

Introduction

Reinforcement Learning (RL) policies often operate as "black box" models, where validation is typically limited to observing average rewards or performance on standard tasks. However, high-performing agents can harbor dangerous or nonsensical behaviors in states rarely encountered during training. To address this, the authors propose REACT (Revealing Evolutionary Action Consequence Trajectories), a framework designed to automatically discover diverse and uncertain "edge-case" scenarios that reveal the true boundaries and hidden flaws of a learned policy.

The REACT Framework

REACT treats the search for informative demonstrations as an optimization problem solved via an Evolutionary Algorithm (EA). Instead of modifying the policy itself, REACT evolves a population of initial states to identify conditions that produce the most revealing trajectories.

The REACT framework uses an evolutionary loop to optimize initial states, generating trajectories that are evaluated by a joint fitness function to provide informative demonstrations.

The Joint Fitness Function

The core innovation of REACT is its joint fitness function, which optimizes for three distinct criteria:

  1. Local Diversity: Prevents repetitive or circular paths by measuring environment coverage within a single trajectory.
  2. Global Diversity: Ensures the human observer sees a wide variety of behaviors by comparing a path to others in the current pool.
  3. Action Certainty: Specifically seeks out states where the agent is uncertain (low probability for the chosen action). These "confused" states often highlight the boundaries of the training data and reveal where the policy is most likely to fail.

Experimental Evaluation

The researchers evaluated REACT across three environments: FlatGrid11 (a baseline gridworld), HoleyGrid11 (a gridworld with hazardous holes), and FetchReach (a continuous 3D robotic control task).

In these environments, REACT demonstrated a superior ability to map the behavioral repertoire of agents compared to random sampling. By specifically targeting areas of low action certainty, the framework successfully identified failure modes in "successfully trained" policies that standard validation methods missed.

Key Results and Insights

Enhanced Environment Coverage

REACT generated a significantly more widespread distribution of trajectories than random sampling. In gridworld tasks, this allowed for a more comprehensive assessment of how an agent navigates different regions of the state space.

Comparison of environment coverage between random sampling and REACT in the FlatGrid11 environment, showing REACT's ability to achieve more diverse and widespread trajectories.

Revealing Overfitting and Policy Brittleness

In the robotic FetchReach task, REACT revealed that longer training does not always equate to better behavior. While a 5M-step policy achieved higher rewards, REACT discovered it was more prone to extreme, jerky movements in edge cases compared to a 3M-step policy. This suggests that REACT can detect behavioral overfitting that remains hidden during standard testing.

Failure Discovery

In the HoleyGrid environment, REACT evolved initial states that led policies to fall into hazards. This capability allows developers to distinguish between policies that are truly robust and those that are merely precise within a narrow range of familiar states.

Limitations and Future Work

Currently, REACT focuses on disturbing initial positions rather than altering environment layouts (e.g., moving walls). Future research could expand the evolutionary search to include environmental perturbations. Additionally, while REACT is designed for human interpretability, formal user studies are needed to quantify its impact on human trust and the ability to predict agent behavior. Finally, the framework holds potential as an adversarial curriculum generator to harden policies against the very failures it discovers.

Summary updated

BibTeX
@inproceedings{altmann2024react,
  author = {Philipp Altmann and Céline Davignon and Maximilian Zorn and Fabian Ritz and Claudia Linnhoff-Popien and Thomas Gabor},
  title = {{REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning}},
  booktitle = {IJCCI},
  year = {2024},
  url = {https://openreview.net/forum?id=DAOtvnToNI},
}
Figure