Reinforcement Learning of Causal Variables Using Mediation Analysis

Tue Herlau, Rasmus LarsenAAAI 2022 · DOI

Abstract

Standard reinforcement learning (RL) focuses on maximizing cumulative rewards, often ignoring the underlying causal structure of the environment. This paper proposes a novel framework for RL agents to autonomously discover abstract causal variables by maximizing the Natural Indirect Effect (NIE) from mediation analysis. By identifying features that are both manipulable by the agent and predictive of future success, the approach enables the learning of interpretable world models. Experiments in grid-world environments demonstrate that this method successfully identifies key causal bottlenecks, such as opening a door to reach a goal, leading to improved performance and more parsimonious representations.

Introduction

In complex environments, RL agents often struggle with generalization because they treat all sensory inputs with equal importance. In contrast, humans utilize causal abstractions—high-level concepts like "the door is locked" or "I have the key"—to navigate the world efficiently. Existing methods for integrating causality into RL typically rely on pre-defined variables or generative models that attempt to predict every detail of the environment. This paper addresses the challenge of discovering these "descriptive" causal variables from scratch, focusing on features that act as a bottleneck between high-level policy choices and final rewards.

Methods

The core innovation of this work is the replacement of the standard RL objective with one derived from mediation analysis. Specifically, the agent seeks to maximize the Natural Indirect Effect (NIE). In this framework, the agent's policy choice (treatment) affects the outcome (reward) through a learned mediating variable $Z$.

Mediation analysis framework for RL, where a policy influences reward through a learned mediating variable Z.

To maximize the NIE, the discovered variable $Z$ must satisfy two conditions:

  1. Manipulability: The agent must be able to learn a policy that reliably makes $Z$ true.
  2. Predictivity: Making $Z$ true must significantly increase the expected reward compared to when $Z$ is false.

The authors derive modified Bellman-like recursive equations to estimate these values and utilize V-trace estimators for efficient off-policy learning from a replay buffer of past experiences.

Experimental Setup

The approach was evaluated on two primary tasks:

  • TWOSTAGE: A synthetic Markov Reward Process designed as a bottleneck environment to validate mathematical convergence.
  • DOORKEY: A grid-world environment where an agent must find a key, unlock a door, and reach a goal. This task is challenging for standard RL due to sparse rewards and the necessity of a specific causal sequence.
A general causal agent interacting with the DOORKEY environment using an internal causal model to achieve policy goals.

Key Insights and Results

The "Causal Learner" successfully identified binary variables correlating with critical milestones, such as whether the door was opened. In the DOORKEY environment, the agent achieved an NIE of 0.098, far surpassing a cross-entropy baseline (0.011) which failed to find manipulable variables.

Crucially, the discovered causal knowledge translated into functional performance. The joint policy derived from the causal model achieved a total reward of 0.300, compared to 0.240 for the baseline policy. This demonstrates that maximizing the NIE naturally steers the agent toward meaningful milestones while ignoring trivial features that are predictive but not actionable (e.g., being one step from the goal).

Discussion and Future Work

While promising, the current framework is limited to single, binary mediating variables. Future research will explore scaling this mediation-based discovery to multiple, continuous, and interconnected variables in high-dimensional environments. Additionally, reducing the computational overhead of the auxiliary value functions required for the modified Bellman updates remains an important area for optimization.

By framing causal variable discovery as a mediation analysis problem, this work provides a principled way for RL agents to learn abstract, actionable concepts, offering a path toward more interpretable and robust artificial intelligence.

Summary updated

BibTeX
@inproceedings{herlau2022reinforcement,
  author = {Tue Herlau and Rasmus Larsen},
  title = {{Reinforcement Learning of Causal Variables Using Mediation Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i6.20648},
}
Figure