Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning

Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo J. Rezende, Yoshua Bengio, Michael Mozer, Christopher J. Pal

Introduction

In traditional causal discovery, researchers typically assume that causal variables are provided as clean, structured data. However, an autonomous agent in a real-world environment must perform causal induction: it must learn high-level representations directly from raw pixels while simultaneously uncovering the causal "rules of the game." This paper introduces a benchmarking suite to evaluate how reinforcement learning (RL) agents discover these structures and compares various world model architectures.

Methodology

The authors evaluate four types of world models within a Model-Based Reinforcement Learning (MBRL) framework:

  • Monolithic Models (AE/VAE): Standard autoencoders that compress the entire visual scene into a single latent vector.
  • Graph Neural Networks (GNNs): Models like C-SWM that represent the world as a collection of objects and learn pairwise interactions.
  • Modular Transition Models (MTM): Architectures that factorize representations for each object and learn directed rules to model how one object's state influences another.

These models are trained using either Maximum Likelihood (NLL) (pixel-level prediction) or Contrastive Learning (state discrimination).

Experimental Framework

The evaluation utilizes two new environment suites designed for parametric control of causal complexity:

  1. Physics (Weighted-block pushing): Blocks have hidden weights; heavier blocks can push lighter ones. Agents must infer this hierarchy.
  2. Chemistry: Objects change colors based on a Directed Acyclic Graph (DAG), where changing a "parent" object triggers a chain reaction in its "children."
Visualizing causal structures through observations and interventions.

Datasets and Complexity

The benchmarking suite allows researchers to manipulate the number of variables, the sparsity of the causal graph, and the length of causal chains. This systematic control reveals how different architectures handle increasing complexity.

Parametric control of graph size and sparsity in the evaluation suite.

Key Insights and Results

  • The Pairwise Limitation of GNNs: While GNNs excel at simple pairwise interactions, they struggle with long causal chains (e.g., A causes B, which causes C). Modular models handle these structures more effectively due to their flexible directed relationship modeling.
  • Scalability: Modular models scale significantly better than monolithic ones. In environments with 5 objects, Modular models achieved nearly double the success rate of VAEs in downstream RL tasks.
  • Zero-Shot Transfer: Structured models demonstrate superior transfer capabilities. For instance, a modular model that learns a weight-color relationship can immediately apply that logic to novel colors it has never encountered.
  • Evaluation Metrics: A key finding is that common metrics like "Hits at Rank 1" do not always correlate with actual RL performance. High prediction accuracy does not necessarily translate to effective planning.

Conclusion

The study demonstrates that modular, object-factored architectures are critical for effective causal discovery from visual data. Future research should focus on continuous interventions and handling unobserved confounders to bridge the gap toward real-world robotics.

Summary updated

Abstract

Inducing causal relationships from observations is a classic problem in machine learning. Most work in causality starts from the premise that the causal variables themselves are observed. However, for AI agents such as robots trying to make sense of their environment, the only observables are low-level variables like pixels in images. To generalize well, an agent must induce high-level variables, particularly those which are causal or are affected by causal variables. A central goal for AI and causality is thus the joint discovery of abstract representations and causal structure. However, we note that existing environments for studying causal induction are poorly suited for this objective because they have complicated task-specific causal graphs which are impossible to manipulate parametrically (e.g., number of nodes, sparsity, causal chain length, etc.). In this work, our goal is to facilitate research in learning representations of high-level variables as well as causal structures among them. In order to systematically probe the ability of methods to identify these variables and structures, we design a suite of benchmarking RL environments. We evaluate various representation learning algorithms from the literature and find that explicitly incorporating structure and modularity in models can help causal induction in model-based reinforcement learning.
BibTeX
@inproceedings{ke2021systematic,
  author = {Nan Rosemary Ke and Aniket Didolkar and Sarthak Mittal and Anirudh Goyal and Guillaume Lajoie and Stefan Bauer and Danilo J. Rezende and Yoshua Bengio and Michael Mozer and Christopher J. Pal},
  title = {{Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=gWIbXsrtOCc},
}
Figure