Abstract
Standard reinforcement learning (RL) focuses on maximizing cumulative rewards, often ignoring the underlying causal structure of the environment. This paper proposes a novel framework for RL agents to autonomously discover abstract causal variables by maximizing the Natural Indirect Effect (NIE) from mediation analysis. By identifying features that are both manipulable by the agent and predictive of future success, the approach enables the learning of interpretable world models. Experiments in grid-world environments demonstrate that this method successfully identifies key causal bottlenecks, such as opening a door to reach a goal, leading to improved performance and more parsimonious representations.
Introduction
In complex environments, RL agents often struggle with generalization because they treat all sensory inputs with equal importance. In contrast, humans utilize causal abstractions—high-level concepts like "the door is locked" or "I have the key"—to navigate the world efficiently. Existing methods for integrating causality into RL typically rely on pre-defined variables or generative models that attempt to predict every detail of the environment. This paper addresses the challenge of discovering these "descriptive" causal variables from scratch, focusing on features that act as a bottleneck between high-level policy choices and final rewards.
Methods
The core innovation of this work is the replacement of the standard RL objective with one derived from mediation analysis. Specifically, the agent seeks to maximize the Natural Indirect Effect (NIE). In this framework, the agent's policy choice (treatment) affects the outcome (reward) through a learned mediating variable $Z$.
To maximize the NIE, the discovered variable $Z$ must satisfy two conditions:
- Manipulability: The agent must be able to learn a policy that reliably makes $Z$ true.
- Predictivity: Making $Z$ true must significantly increase the expected reward compared to when $Z$ is false.
The authors derive modified Bellman-like recursive equations to estimate these values and utilize V-trace estimators for efficient off-policy learning from a replay buffer of past experiences.
Experimental Setup
The approach was evaluated on two primary tasks:
- TWOSTAGE: A synthetic Markov Reward Process designed as a bottleneck environment to validate mathematical convergence.
- DOORKEY: A grid-world environment where an agent must find a key, unlock a door, and reach a goal. This task is challenging for standard RL due to sparse rewards and the necessity of a specific causal sequence.
Key Insights and Results
The "Causal Learner" successfully identified binary variables correlating with critical milestones, such as whether the door was opened. In the DOORKEY environment, the agent achieved an NIE of 0.098, far surpassing a cross-entropy baseline (0.011) which failed to find manipulable variables.
Crucially, the discovered causal knowledge translated into functional performance. The joint policy derived from the causal model achieved a total reward of 0.300, compared to 0.240 for the baseline policy. This demonstrates that maximizing the NIE naturally steers the agent toward meaningful milestones while ignoring trivial features that are predictive but not actionable (e.g., being one step from the goal).
Discussion and Future Work
While promising, the current framework is limited to single, binary mediating variables. Future research will explore scaling this mediation-based discovery to multiple, continuous, and interconnected variables in high-dimensional environments. Additionally, reducing the computational overhead of the auxiliary value functions required for the modified Bellman updates remains an important area for optimization.
By framing causal variable discovery as a mediation analysis problem, this work provides a principled way for RL agents to learn abstract, actionable concepts, offering a path toward more interpretable and robust artificial intelligence.
Summary updated