Navigating causal deep learning

Jeroen Berrevoets, Krzysztof Kacprzyk, Zhaozhi Qian, Mihaela van der SchaararXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a "map of causal deep learning" (CDL) to organize a rapidly growing and often confusing field. By refining Pearl’s ladder of causation and adding a second dimension for parametric assumptions, the authors provide a framework to categorize models by their structural knowledge and functional flexibility, helping researchers identify "white space" in the literature and practitioners select the right tools for their data.

The Problem: An Avalanche of Research

Deep learning (DL) excels at finding patterns in high-dimensional data but often lacks the ability to reason about cause and effect. Conversely, traditional causal inference provides rigorous frameworks like Structural Causal Models (SCMs) but often relies on simple, rigid functional forms.

The emerging field of Causal Deep Learning (CDL) attempts to marry these two worlds. However, the field is currently an "avalanche" of papers that are difficult to compare. A Transformer-based model and a linear regression model might both address "association," but they sit at opposite ends of the complexity spectrum. Existing frameworks, like Judea Pearl’s "ladder of causation," focus solely on causal rungs (association, intervention, counterfactuals) and ignore the "deep" part of deep learning—the parametric flexibility and representation space.

Refining the Ladder of Causation into the Map of CDL.

The Two-Axis Framework

The authors propose a two-axis "map" to categorize any CDL method based on its assumptions and capabilities:

  1. The Structural Scale (Vertical Axis): Measures how much we know about the relationships between variables.
  2. Level 1 (Unknown): No prior knowledge; variables are assumed to be potentially dependent.
  3. Level 2 (Plausible): Knowledge of statistical independencies (e.g., Probabilistic Graphical Models), but exact causal direction might be unknown. The authors call this "Rung 1.5."
    • Level 3 (Causal): The exact causal structure (DAG) is known.
  4. The Parametric Scale (Horizontal Axis): Measures the assumptions made about the functions connecting variables.
  5. Non-parametric: No assumptions on functional form (the strength of Deep Learning).
  6. Noise Models: Assumptions on how noise enters the system (e.g., Additive Noise Models).
  7. Fully Parametric: Specific forms like linear or $n$-times differentiable functions.
  8. Fully Known: The functions are already known (no learning required).

The Map of CDL

The "Map" applies these scales to both the Input (what the model assumes) and the Representation (what the model learns or outputs). This distinction is crucial: a model might take non-parametric data as input but output a highly structured causal representation.

The complete Map of CDL framework showing the dual-grid system for inputs and representations.

Key Insights and Research Gaps

  • The "Rung 1.5" Concept: The authors argue that Pearl’s first rung (Association) is too broad for machine learning. They distinguish between models that assume nothing and models that encode statistical independencies, placing the latter at a hypothetical "Rung 1.5."
  • The Pillars of CDL: CDL is defined by the interaction between leveraging structure (causality's strength) and non-parametric estimation (deep learning's strength). These are often in conflict; more structure usually requires stricter parametric assumptions to remain identifiable.
  • Identifiability vs. Flexibility: The map visually demonstrates that the "holy grail"—mapping completely unknown, non-parametric input to a fully causal, non-parametric representation—is widely considered impossible without some form of inductive bias.
  • Identifying Research Gaps: By plotting existing papers, the authors show that certain areas are overcrowded (e.g., learning structure in additive noise settings), while others are virtually empty (e.g., fully known parametric models, which don't require learning).

Conclusion

The primary result is the Map of CDL itself. The authors demonstrate its utility by categorizing high-profile supervised learning papers and framing the work as a "living paper." They intend to host an online forum where the community can discuss and propose changes to the map as the field evolves, providing a much-needed navigation system for the future of causal AI.

Summary updated

Abstract

Causal deep learning (CDL) is a new and important research area in the larger field of machine learning. With CDL, researchers aim to structure and encode causal knowledge in the extremely flexible representation space of deep learning models. Doing so will lead to more informed, robust, and general predictions and inference -- which is important! However, CDL is still in its infancy. For example, it is not clear how we ought to compare different methods as they are so different in their output, the way they encode causal knowledge, or even how they represent this knowledge. This is a living paper that categorises methods in causal deep learning beyond Pearl's ladder of causation. We refine the rungs in Pearl's ladder, while also adding a separate dimension that categorises the parametric assumptions of both input and representation, arriving at the map of causal deep learning. Our map covers machine learning disciplines such as supervised learning, reinforcement learning, generative modelling and beyond. Our paradigm is a tool which helps researchers to: find benchmarks, compare methods, and most importantly: identify research gaps. With this work we aim to structure the avalanche of papers being published on causal deep learning. While papers on the topic are being published daily, our map remains fixed. We open-source our map for others to use as they see fit: perhaps to offer guidance in a related works section, or to better highlight the contribution of their paper.
BibTeX
@misc{berrevoets2022navigating,
  author = {Jeroen Berrevoets and Krzysztof Kacprzyk and Zhaozhi Qian and Mihaela van der Schaar},
  title = {{Navigating causal deep learning}},
  year = {2022},
  eprint = {2212.00911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.00911},
}
Figure