Abstract
We employ an evolutionary optimization framework that perturbs initial states to generate informative and diverse policy demonstrations. A joint surrogate fitness function guides the optimization by combining local diversity, behavioral certainty, and global population diversity. To assess demonstration quality, we apply a set of evaluation metrics, including the reward-based optimality gap, fidelity interquartile means (IQMs), fitness composition analysis, and trajectory visualizations. Hyperparameter sensitivity is also examined to better understand the dynamics of trajectory optimization. Our findings demonstrate that optimizing trajectory selection via surrogate fitness metrics significantly improves interpretability of RL policies in both discrete and continuous environments. In gridworld domains, evaluations reveal significantly enhanced demonstration fidelities compared to random and ablated baselines. In continuous control, the proposed framework offers valuable insights, particularly for early-stage policies, while fidelity-based optimization proves more effective for mature policies. By refining and systematically analyzing surrogate fitness functions, this study advances the interpretability of RL models. The proposed improvements provide deeper insights into RL decision-making, benefiting applications in safety-critical and explainability-focused domains.
Introduction
Deep Reinforcement Learning (RL) policies are notoriously opaque "black boxes." Traditionally, these agents are evaluated based on their ability to maximize cumulative rewards. However, a high score fails to explain the agent's underlying logic or its potential for failure in novel situations. To establish trust in safety-critical applications, it is necessary to move beyond "best-case" scenarios and identify where agents hesitate, oscillate, or encounter the "long tail" of rare edge cases. Manually identifying these informative states in high-dimensional spaces is often infeasible.
Methodology: The REACT Framework
To address this, we introduce REACT (Revealing Evolutionary Action Consequence Trajectories), a model-agnostic framework that treats the search for informative demonstrations as an optimization problem.
Instead of training the policy itself, REACT utilizes an Evolutionary Algorithm (EA) to evolve a population of initial states. The fitness of these states is determined by a Joint Surrogate Fitness Function composed of three critical components:
- Local Diversity ($D_l$): Measures the variety of states visited within a single trajectory.
- Global Diversity ($D_g$): Ensures the trajectory provides information distinct from other demonstrations in the current pool.
- Certainty ($C$): Identifies regions where the policy exhibits high entropy (low confidence). These low-certainty regions often reveal the decision boundaries of the agent's knowledge.
By maximizing this joint fitness, REACT systematically populates a demonstration pool with trajectories that are both diverse and highly descriptive of the policy's underlying behavior.
Experimental Setup
The framework was evaluated across three distinct environments to test its versatility:
- FlatGrid11: A standard 11x11 discrete gridworld for baseline testing.
- HoleyGrid11: A complex gridworld containing traps that terminate episodes with a penalty, testing obstacle navigation.
- FetchReach: A continuous robotic control task involving a 6-Degree-of-Freedom arm, testing performance in high-dimensional state/action spaces.
Key Results and Insights
We utilized the Fidelity Interquartile Mean (IQM) to quantitatively assess how well the generated trajectories represent the policy's overall behavior space.
Quantitative Performance
| Environment | REACT (Ours) | Random Search | Training State |
|---|---|---|---|
| FlatGrid11 | 0.744 ± 0.147 | 0.355 ± 0.120 | 0.000 ± 0.000 |
| HoleyGrid11 | 2.130 ± 0.324 | 1.486 ± 0.250 | 0.000 ± 0.000 |
| FetchReach (Early) | 0.842 ± 0.127 | 0.106 ± 0.025 | 0.000 ± 0.000 |
Diversity vs. Fidelity
A key finding was that directly optimizing for "fidelity" (how well a trajectory represents average behavior) often resulted in less informative summaries. Conversely, optimizing for diversity and uncertainty (surrogates) was significantly more effective at uncovering the full range of policy behaviors. In HoleyGrid11, REACT-generated demonstrations showed a higher density of visits to challenging "outer-state" regions compared to random sampling.
Policy Maturity
The effectiveness of these metrics depends on policy maturity. For "young," early-stage policies, REACT's diversity-focused approach is vastly superior. For highly mature, converged policies, direct fidelity optimization becomes more competitive, though REACT still provides superior spatial coverage of the environment.
Conclusion and Future Work
REACT demonstrates that evolutionary optimization of surrogate fitness metrics can effectively "crack" the RL black box by discovering informative edge cases. Currently, the framework focuses on perturbing the initial state. Future research will explore perturbing the environment itself—such as moving obstacles or altering physics—to identify robust failure modes. Additionally, integrating REACT into the training loop as an automated adversarial curriculum could help agents learn more robustly from the outset.
Summary updated