Abstract
Deep Reinforcement Learning (DRL) has demonstrated significant potential in complex decision-making tasks but remains limited by its "black box" nature and sample inefficiency. We present ERCI (Experience Replay with Causal Inference), a novel framework that integrates causal reasoning into the DRL training loop. By treating agent experiences as sequences of causal events rather than isolated transitions, ERCI identifies high-impact factors that drive success, prioritizing them for learning while providing human-readable explanations for agent behavior.
Introduction
Despite the success of DRL in various domains, two critical challenges persist: inscrutability and sample inefficiency. In safety-critical applications like autonomous driving, it is vital to understand why an agent makes a specific decision. Furthermore, standard Experience Replay (ER) techniques often treat all past transitions as equally important, failing to capture the underlying causal logic of the environment. ERCI addresses these gaps by leveraging causal discovery to prioritize meaningful experiences and generate transparent decision-making models.
The ERCI Framework
The ERCI framework introduces a three-module pipeline designed to inject causal reasoning directly into the data sampling process of DRL.
- Time Series Representation: Raw interaction data is segmented using a two-stage clustering approach (TICC and K-Means) into discrete Time Series Causal Factors (TSCFs). These factors represent meaningful behavioral patterns, such as "accelerating" or "turning left."
- Internal Causality Discovery: The system employs the Greedy Fast Causal Inference (GFCI) algorithm to construct a Partial Ancestral Graph (PAG). This graph maps the relationships between TSCFs and the final reward. A "Time Correction" step ensures that the discovered causal links respect temporal order.
- Explainable DRL Model: The "Causal Strength" derived from the discovery phase is used to prioritize sampling from the experience replay buffer. A Causal Replay Weight Tree manages these priorities, focusing the agent's learning on experiences with strong causal links to high rewards.
Experimental Evaluation
ERCI was evaluated across several demanding environments, including Highway-Env for autonomous driving, Gym (MuJoCo) for robotics benchmarks, and the CARLA Simulator for high-fidelity overtaking scenarios.
Key Results
- Performance Improvement: ERCI significantly outperformed state-of-the-art baselines, improving the Average Score (AS) of the DDPG algorithm by 20.70% across tested environments.
- Faster Convergence: The framework reached target performance levels more quickly than standard models, demonstrating superior sample efficiency.
- Scalability: ERCI was successfully integrated with Prioritized Experience Replay (PER) to create "PERCI," showing consistent performance gains and compatibility with existing optimization techniques.
Explainability and Human-Aligned Insights
One of the primary advantages of ERCI is its ability to provide human-aligned explanations. Because TSCFs are clustered into distinct, recognizable patterns, the resulting causal graphs are interpretable. For example, in an overtaking scenario, the model can explicitly identify that "turning left" and "accelerating" were the direct causes of avoiding an accident or reaching a goal.
Conclusion
ERCI bridges the gap between causal inference and deep reinforcement learning by using causality as a robust sampling metric. While the framework introduces some computational overhead due to the discovery process, it offers a significant leap forward in making DRL agents both more efficient and more transparent. Future work will focus on refining clustering sensitivity and handling unobserved confounding variables in even more complex real-world robotic tasks.
Summary updated