A Neural Mean Embedding Approach for Back-door and Front-door Adjustment

Liyuan Xu, Arthur Gretton

Causal inference is often described as the "science of the 'what if'." While standard machine learning excels at predicting outcomes based on patterns, causal inference aims to understand the effect of interventions—like how a specific medical treatment changes a patient's life expectancy. The challenge? Hidden "confounders" (like genetics or social status) that influence both the treatment and the outcome, muddying the waters of our data.

In their paper, "A Neural Mean Embedding Approach for Back-door and Front-door Adjustment," Liyuan Xu and Arthur Gretton introduce a powerful new framework that combines the structural rigor of Pearl’s causal calculus with the representational power of deep learning.

TL;DR

The authors propose a two-stage neural network approach to estimate causal effects (ATE and ATT) using back-door and front-door adjustment. By replacing fixed, hand-crafted kernels with adaptive "neural mean embeddings," the method can handle complex, high-dimensional data—like images—where traditional causal methods typically fail.

What Problem Does This Paper Address?

In an ideal world, we’d run randomized controlled trials (RCTs) for everything. In the real world, we often only have observational data. To extract causal truth from this data, we use "adjustments":

  • Back-door adjustment: Using observed covariates (like symptoms) to block the influence of hidden confounders (like genetics) on the treatment.
  • Front-door adjustment: Using a mediator variable (like the amount of tar in a smoker's lungs) that sits between the treatment and the outcome to isolate the causal path.
Causal graphs illustrating general, back-door, and front-door adjustments.

The technical hurdle is that these adjustments require calculating expectations over conditional distributions. Traditional methods either rely on density estimation (which breaks down in high dimensions) or fixed feature maps (which aren't flexible enough for complex data like images). This paper seeks to bridge that gap.

Methods

The authors propose a two-stage regression procedure powered by neural networks:

  1. Stage 1 (The "What Happens" Regression): They learn a conditional mean function $g(a, x)$ that predicts the outcome $Y$ given treatment $A$ and covariate $X$. The "trick" here is the model architecture: they use a tensor product of neural features ($\phi_A(a) \otimes \phi_X(x)$) with a linear final layer. This specific structure allows them to separate the treatment's features from the covariate's features.
  2. Stage 2 (The "What If" Adjustment): To get the causal effect, they need to average the Stage 1 function over the distribution of the covariates. Instead of sampling or estimating densities, they compute Neural Mean Embeddings. They essentially train a second neural network to predict the features of the first stage.

Because the final layer of Stage 1 is linear, the expectation of the function is simply the function applied to the expectation of the features. This elegant mathematical shortcut avoids the "curse of dimensionality" that plagues density-based methods.

Datasets

The authors evaluated their approach on three distinct benchmarks:

  • IHDP (Infant Health and Development Program): A standard semi-synthetic dataset for binary treatments.
  • ACIC (Atlantic Causal Inference Conference): A more challenging benchmark characterized by "extreme propensity scores" (where some people are much more likely to get a treatment than others), which often breaks standard estimators.
  • dSprite: A high-dimensional image dataset. Here, the "treatment" is an actual image of a shape, requiring the model to extract causal signals from 4,096-dimensional pixel data.

Key Insights and Ideas

  • Adaptive vs. Fixed Features: Previous state-of-the-art methods used fixed kernels (RKHS). This paper shows that letting the neural network learn the features specifically for the causal task leads to much better performance, especially when the data is unstructured (like images).
  • The Power of the Tensor Product: By forcing the first-stage regression into a tensor product form, the authors make the subsequent integration (the "adjustment") computationally trivial. It turns a hard integration problem into a simple linear algebra problem.
  • Theoretical Consistency: The authors don't just show it works empirically; they provide a consistency proof using Rademacher complexity, showing that as you get more data, the model's estimate converges to the true causal parameter.

Key Results

Performance comparison on IHDP and ACIC datasets showing the proposed NN Embedding method outperforming baselines.
  • Superiority on ACIC: On the challenging ACIC dataset, the proposed "NN Embedding" achieved an ATE error of 0.231, significantly outperforming RieszNet (1.071) and RKHS-based methods (1.785).
  • Handling Images: In the dSprite experiment, where the treatment was an image, the neural approach maintained low error while the fixed-kernel RKHS method struggled to capture the complex relationships within the pixel data.
  • Robustness: The method proved particularly robust in settings with continuous treatments, where many traditional "discretization" methods fail.

Limitations and Future Work

While powerful, the authors note a few trade-offs:

  • Convergence Speed: The theoretical convergence rate is slightly slower than some kernel methods. This is the price paid for the flexibility of assuming only "Lipschitz continuity" rather than the stricter requirements of a Reproducing Kernel Hilbert Space (RKHS).
  • Computational Cost: Training two stages of neural networks, especially with tensor products, can be more intensive than simpler linear models.
  • Future Directions: The authors suggest extending this "adaptive feature" approach to more complex causal scenarios, such as dynamic treatment effects (where treatments change over time) and marginal average effects.

This work represents a significant step forward in making causal inference practical for the "Deep Learning era," proving that we don't have to choose between structural causal integrity and the power of neural representations.

Summary updated

Abstract

We consider the estimation of average and counterfactual treatment effects, under two settings: back-door adjustment and front-door adjustment. The goal in both cases is to recover the treatment effect without having an access to a hidden confounder. This objective is attained by first estimating the conditional mean of the desired outcome variable given relevant covariates (the ``first stage" regression), and then taking the (conditional) expectation of this function as a ``second stage" procedure. We propose to compute these conditional expectations directly using a regression function to the learned input features of the first stage, thus avoiding the need for sampling or density estimation. All functions and features (and in particular, the output features in the second stage) are neural networks learned adaptively from data, with the sole requirement that the final layer of the first stage should be linear. The proposed method is shown to converge to the true causal parameter, and outperforms the recent state-of-the-art methods on challenging causal benchmarks, including settings involving high-dimensional image data.
BibTeX
@inproceedings{xu2023neural,
  author = {Liyuan Xu and Arthur Gretton},
  title = {{A Neural Mean Embedding Approach for Back-door and Front-door Adjustment}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=rLguqxYvYHB},
}
Figure