Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning

Nan Rosemary Ke, Aniket Rajiv Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Jimenez Rezende, Michael Curtis Mozer, Yoshua Bengio, Christopher Pal

TL;DR

This paper tackles the challenge of teaching AI agents to understand cause-and-effect relationships in complex visual environments, a crucial step for robust generalization. They introduce new benchmark environments and find that models with explicit structural and modular components significantly outperform simpler, monolithic models in learning these causal structures.

What Problem Does This Paper Address?

Modern AI agents, especially in reinforcement learning (RL), often operate with raw, low-level sensory inputs like pixels. To truly generalize and act intelligently, these agents need to go beyond mere pattern recognition and infer the underlying causal mechanisms that govern their environment. This means not only learning abstract representations of objects and their properties but also understanding how these abstract variables influence each other (i.e., discovering the causal structure).

Existing benchmarks for studying causal discovery in RL are often too simplistic or have fixed, complex causal structures that are difficult to manipulate. This makes it hard to systematically evaluate and compare different methods for learning causal relationships. The authors aim to bridge this gap by providing a more controlled and modular framework for evaluating causal induction in model-based RL (MBRL).

Methods

The core of this work lies in proposing a new suite of environments designed to isolate and test specific aspects of causal discovery in MBRL. They then evaluate various model architectures on these environments.

The key methodological contribution is the introduction of a new set of benchmarking environments that allow for systematic modulation of causal graph properties (structure, size, sparsity, cause-effect delay). This enables researchers to disentangle the effects of different causal complexities on learning algorithms.

They evaluate two main categories of models:

  • Monolithic Models: These are standard neural network architectures (like autoencoders and variational autoencoders) that lack explicit structural biases beyond their layered design.
  • Modular and Structured Models: These models incorporate explicit architectural components designed to capture relationships between variables. This includes:
  • Graph Neural Networks (GNNs): These models represent variables as nodes and interactions as edges, effectively modeling pairwise relationships and being permutation invariant.
  • Modular Networks: These models explicitly factorize representations and learn directed relationships between variables, aiming to model causal influences more directly.

The evaluation uses a combination of prediction metrics (likelihood, ranking loss) and downstream RL performance.

Datasets

The paper introduces two new suites of environments for evaluating causal induction in MBRL:

  • Physics Environment: Simulates simple physics with blocks of different weights. The causal relationships are based on how heavier blocks push lighter ones. This environment allows for variations in observation (fully observed vs. unobserved weights) and graph complexity.
  • Chemistry Environment: Allows for arbitrary causal graphs based on simple chemical reactions where changing the state (color) of one variable can affect others. This environment offers fine-grained control over graph structure, size, and stochasticity.

These environments are designed to be minimalist and easily visualizable, with a focus on the underlying causal graph.

Key Insights and Ideas

  • Structure is Key for Causal Induction: The most significant finding is that explicitly incorporating structure and modularity into MBRL models is crucial for effective causal discovery, especially in complex environments.
  • Modular Models Scale Better: Modular networks, which learn directed relationships, demonstrate a superior ability to scale to larger and more complex causal graphs compared to other models.
  • GNNs Excel at Pairwise Interactions: While GNNs are effective at modeling pairwise interactions and perform well on simpler graphs, they struggle with complex causal chains.
  • Evaluation Metrics Matter: The paper highlights that standard prediction metrics (like likelihood and ranking loss) don't always correlate well with downstream RL performance, underscoring the need for more comprehensive evaluation.
  • Causality as an Inductive Bias: The authors hypothesize that causal structure can serve as a powerful inductive bias, guiding the learning of more robust and generalizable world models.

Key Results

  • Models with explicit structure (GNNs and modular models) consistently outperform monolithic models on both prediction metrics and downstream RL performance across the Physics and Chemistry environments.
  • Modular models showed a significant advantage in scaling to larger graphs (e.g., 5 objects in the Physics environment) where monolithic models and GNNs experienced performance drops.
  • For the Chemistry environment, modular models achieved the best performance across various causal graph complexities, particularly on chain and full graphs with long cause-effect chains.
  • In the Physics environment, GNNs performed well with 3 variables but their performance degraded with 5 objects, whereas modular models maintained higher performance.
  • Training with a contrastive loss function generally led to better performance for structured models compared to likelihood loss, especially in zero-shot transfer settings.

Limitations and Future Work

  • Evaluation Metric Discrepancy: The authors note that prediction metrics (likelihood, ranking) do not always align with downstream RL performance, suggesting a need for further investigation into appropriate evaluation criteria.
  • Fixed Timescales: The current environments assume fixed timescales for cause-and-effect. The authors suggest extending environments to handle variable timescales, which is highly relevant for credit assignment in RL.
  • Meta-Learning: Exploring meta-learning scenarios where causal graphs change per episode is proposed as a future direction.
  • Unobserved Confounders: While acknowledged as a challenge, the paper focuses on environments with observable variables. Future work could delve deeper into handling unobserved confounders.

Summary updated

Abstract

Inducing causal relationships from observations is a classic problem in machine learning. Most work in causality starts from the premise that the causal variables themselves have known semantics or are observed. However, for AI agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level variables, particularly those which are causal or are affected by causal variables. A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure. In this work, we systematically evaluate the agent's ability to learn underlying causal structure. We note that existing environments for studying causal induction are poorly suited for this objective because they have complicated task-specific causal graphs with many confounding factors. Hence, to facilitate research in learning the representation of high-level variables as well as causal structure among these variables, we present a suite of RL environments created to systematically probe the ability of methods to identify variables as well as causal structure among those variables. We evaluate various representation learning algorithms from literature and found that explicitly incorporating structure and modularity in the model can help causal induction in model-based reinforcement learning.
BibTeX
@inproceedings{ke2021systematic,
  author = {Nan Rosemary Ke and Aniket Rajiv Didolkar and Sarthak Mittal and Anirudh Goyal and Guillaume Lajoie and Stefan Bauer and Danilo Jimenez Rezende and Michael Curtis Mozer and Yoshua Bengio and Christopher Pal},
  title = {{Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=gWIbXsrtOCc},
}
Figure