A Neural Mean Embedding Approach for Back-door and Front-door Adjustment

Liyuan Xu, Arthur Gretton

Abstract

We consider the estimation of average and counterfactual treatment effects, under two settings: back-door adjustment and front-door adjustment. The goal in both cases is to recover the treatment effect without having an access to a hidden confounder. This objective is attained by first estimating the conditional mean of the desired outcome variable given relevant covariates (the "first stage" regression), and then taking the (conditional) expectation of this function as a "second stage" procedure. We propose to compute these conditional expectations directly using a regression function to the learned input features of the first stage, thus avoiding the need for sampling or density estimation. All functions and features (and in particular, the output features in the second stage) are neural networks learned adaptively from data, with the sole requirement that the final layer of the first stage should be linear. The proposed method is shown to converge to the true causal parameter, and outperforms the recent state-of-the-art methods on challenging causal benchmarks, including settings involving high-dimensional image data.

Introduction

The primary objective of causal inference is to estimate the effect of an action (treatment) on an outcome using observational data. This task is complicated by the presence of confounders—unobserved variables that influence both the treatment and the outcome. To address this, researchers employ "adjustments" based on the underlying causal structure.

Causal graphs for (a) general case, (b) back-door adjustment, and (c) front-door adjustment.

The back-door adjustment blocks non-causal paths from confounders to treatments using measurable covariates $X$. The front-door adjustment utilizes an intermediate variable $M$ that mediates the effect of treatment $A$ on outcome $Y$. While these theoretical frameworks are well-established, implementing them for high-dimensional or continuous data remains challenging. Traditional methods often rely on density estimation, which struggles in high dimensions, or kernel methods that use fixed, non-adaptive features.

Methods: Neural Mean Embedding

This paper proposes a two-stage regression framework called Neural Mean Embedding that leverages the flexibility of deep learning to learn adaptive feature representations.

  1. Stage 1 (Feature Learning): A neural network is trained to predict the outcome $Y$ from treatment $A$ and covariates $X$. The architecture is constrained such that the final layer is a linear combination of non-linear features learned by preceding layers: $g(a, x) = w^T (\phi_A(a) \otimes \phi_X(x))$.
  2. Stage 2 (Mean Embedding): Due to the linearity of the final layer in Stage 1, the expectation of the outcome (the causal effect) can be computed by finding the expectation of the learned features. Instead of complex density estimation, a second neural network is trained to directly learn these feature means from the data.

By using neural networks for both stages, the model learns "adaptive feature dictionaries," identifying the most relevant data aspects for causal estimation.

Results and Discussion

The method was evaluated on standard benchmarks like the Infant Health and Development Program (IHDP) and synthetic high-dimensional datasets like dSprite (images).

MSE comparison of the proposed method against state-of-the-art algorithms on the dSprite dataset.

On the dSprite image dataset, the Neural Mean Embedding approach significantly outperformed state-of-the-art methods such as ReiszNet and RKHS Embedding, achieving substantially lower Mean Squared Error (MSE). On the lower-dimensional IHDP dataset, the method remained competitive with specialized models like DragonNet.

Key insights include:

  • Linearity as a Tool: Forcing linearity in the final layer of the first stage allows the expectation operator to be "pushed" inside the function, bypassing the curse of dimensionality.
  • Adaptive Features: Learning representations via neural networks is far more effective for complex data (like images) than using fixed kernel features.
  • Theoretical Consistency: The authors provide a formal consistency proof showing that the model's estimate converges to the true causal parameter as data size increases.

Conclusion

The Neural Mean Embedding framework provides a unified, flexible approach for back-door and front-door adjustments. By combining the theoretical strengths of mean embeddings with the representational power of deep learning, it enables robust causal inference in high-dimensional settings where traditional methods fail. Future work may extend this logic to marginal average effects or dynamic treatment regimes.

Summary updated

Abstract

We consider the estimation of average and counterfactual treatment effects, under two settings: back-door adjustment and front-door adjustment. The goal in both cases is to recover the treatment effect without having an access to a hidden confounder. This objective is attained by first estimating the conditional mean of the desired outcome variable given relevant covariates (the "first stage" regression), and then taking the (conditional) expectation of this function as a "second stage" procedure. We propose to compute these conditional expectations directly using a regression function to the learned input features of the first stage, thus avoiding the need for sampling or density estimation. All functions and features (and in particular, the output features in the second stage) are neural networks learned adaptively from data, with the sole requirement that the final layer of the first stage should be linear. The proposed method is shown to converge to the true causal parameter, and outperforms the recent state-of-the-art methods on challenging causal benchmarks, including settings involving high-dimensional image data.
BibTeX
@inproceedings{xu2023neural,
  author = {Liyuan Xu and Arthur Gretton},
  title = {{A Neural Mean Embedding Approach for Back-door and Front-door Adjustment}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=rLguqxYvYHB},
}
Figure