Explanation-Aware Experience Replay in Rule-Dense Environments

Francesco Sovrano, Alex Raymond, Amanda ProrokIEEE RA-L 2022 · DOI

Introduction

Reinforcement Learning (RL) agents frequently encounter difficulties when transitioning from simplified simulations to real-world environments characterized by dense, complex rules and numerous edge cases. In domains like autonomous driving, an agent must adhere to varying speed limits, weather-dependent regulations, and specific vehicle class constraints. Standard approaches, such as Prioritized Experience Replay (PER), sample past experiences based on Temporal Difference (TD) error—a measure of "surprise." However, PER lacks semantic understanding; it cannot distinguish between a rare crash and a rare high-speed maneuver if their error magnitudes are similar. Furthermore, hard-coding rules into the observation space often results in brittle systems that require extensive retraining when regulations change.

To address these challenges, this paper introduces Explanation-Aware Experience Replay (XAER). XAER partitions an agent's memory into clusters based on semantic "explanations" (e.g., "speeding" or "off-road"), allowing the agent to prioritize rare but critical corner cases and develop a more robust policy in rule-dense environments.

Methods: The XAER Framework

The core innovation of XAER is the transformation of rule-based explanations into a case-based learning mechanism. Unlike a standard "flat" memory buffer, XAER utilizes an "Explainer" module to label state transitions and organize them into distinct clusters ($C_e$).

The XAER architecture: An Explainer module appends semantic labels to state transitions, which are then stored in clustered experience replay buffers for prioritized sampling.

The framework operates through three primary clustering strategies:

  1. HOW explanations: Clusters are based on performance metrics (e.g., "performing above average" vs. "below average").
  2. WHY explanations: Clusters are based on semantic reasons for outcomes (e.g., "penalty for speeding," "goal reached").
  3. HOW+WHY: A hybrid approach combining both performance and semantic labels.

Sampling in XAER involves a two-tier prioritization strategy. At the inter-cluster level, the agent selects which type of experience to sample, often following a "simplicity" heuristic that prioritizes simpler explanations to create a natural learning curriculum. At the intra-cluster level, standard TD-error is used to select the most informative transitions within the chosen cluster. A hyperparameter $\xi$ (cluster size proportion) controls the balance between over-sampling rare events and following the real-world distribution, preventing the agent from over-fitting to rare exceptions.

Benchmarking in Rule-Dense Environments

To evaluate XAER, the authors developed two modular benchmarks designed specifically for testing rule-heavy scenarios: GridDrive and GraphDrive.

Visualizations of the GridDrive (left) and GraphDrive (right) benchmarks. GridDrive uses a discrete grid with dynamic speed limits, while GraphDrive features continuous kinematics on a planar graph.
  • GridDrive (Discrete): A 15x15 grid where cells represent different road types (e.g., motorway, school zone) with modifiers like weather or accidents. Agents must navigate to a goal while respecting dynamic speed limits.
  • GraphDrive (Continuous): A more complex environment utilizing a planar graph and Ackermann-based vehicle kinematics. Agents must manage steering and acceleration while adhering to road rules.

Both environments were tested across Easy (5 explanations), Medium (12), and Hard (20) rulesets to assess how agents scale with increasing regulatory complexity.

Key Insights and Results

The integration of XAER into off-policy algorithms like DQN, TD3, and SAC yielded significant performance improvements over standard PER baselines.

  • Performance Gains: In the "GridDrive Hard" task, XADQN (XAER-enhanced DQN) achieved 81% higher rewards than the baseline.
  • Robustness in Sparse Scenarios: In the most challenging "GraphDrive Hard" scenario with sparse rewards, standard TD3 and SAC baselines failed to learn a functional policy. In contrast, XASAC successfully navigated the environment and achieved positive results.
  • Agnostic Architecture: By keeping explanations within the replay buffer rather than the input vector, the agent's neural network remains agnostic to specific rules. This makes the system more robust to regulatory changes, as the agent learns the semantic meaning of categories through association rather than explicit architectural constraints.

Conclusion and Future Work

XAER demonstrates that "explanation engineering"—providing semantic labels for transitions—can be more effective than traditional reward engineering in complex environments. By leveraging a curriculum of simplicity, agents can master basic navigation before tackling intricate exceptions.

Future research will explore the application of XAER to on-policy algorithms like PPO and investigate a broader range of explanation types, such as "WHERE," "WHEN," or "WHAT," to further refine the experience clustering process.

Summary updated

BibTeX
@article{sovrano2022explanation,
  author = {Francesco Sovrano and Alex Raymond and Amanda Prorok},
  title = {{Explanation-Aware Experience Replay in Rule-Dense Environments}},
  journal = {IEEE RA-L},
  year = {2022},
  doi = {10.1109/lra.2021.3135927},
}
Figure