Neural Score Matching for High-Dimensional Causal Inference

Oscar Clivio, Fabian Falck, Brieuc Lehmann, George Deligiannidis, Chris Holmes

TL;DR

Neural Score Matching addresses the "curse of dimensionality" in causal inference by matching treated and control units in a multivariate latent space. By leveraging the intermediate layers of a neural network trained to predict propensity scores, the authors provide a flexible way to balance high-dimensional covariates while retaining more structural information than traditional scalar propensity scores.

What Problem Does This Paper Address?

In causal inference, we often want to estimate the effect of a treatment (e.g., a new drug or a policy change) by comparing "treated" individuals to similar "control" individuals. This process, called matching, aims to ensure that the only meaningful difference between groups is the treatment itself, thereby eliminating confounding.

However, as the number of features (covariates) grows, matching becomes incredibly difficult—a phenomenon known as the curse of dimensionality.

  1. Exact matching fails because no two individuals are identical across hundreds of variables.
  2. Mahalanobis distance matching degrades because it treats all covariate interactions as equally important.
  3. Propensity Score Matching (PSM), the standard solution, compresses all information into a single scalar probability. While this avoids the curse of dimensionality, it often discards too much information, potentially matching units that have the same probability of treatment but are otherwise completely unrelated.

The authors seek a "middle ground": a multivariate representation that is low-dimensional enough to match effectively but rich enough to maintain covariate balance.

Methods

The authors introduce Neural Score Matching, a two-stage procedure:

  1. Stage 1: Model Fitting. Train a neural network to predict the treatment assignment ($T$) from the covariates ($X$). This is essentially a propensity score model.
  2. Stage 2: Matching. Instead of matching on the final scalar output (the propensity score), match on the activations of an intermediate hidden layer.
Figure 1: An illustration of neural score matching. In the first stage, a propensity score model is fitted to obtain low-dimensional balancing scores. In the second stage, samples are matched (to one neighbour) in the balancing score space based on a given distance metric. Matched samples are subsequently used to estimate the ATT.

The core innovation lies in the theoretical justification. The authors prove that any intermediate representation in a hierarchical model that eventually predicts the propensity score is itself a balancing score. They show that the imbalance in the original high-dimensional space is mathematically bounded by the imbalance in this lower-dimensional "neural score" space. Specifically, they highlight the utility of using the first hidden layer (before non-linear activations) as a linear balancing score.

Datasets

The method was evaluated on three semi-synthetic benchmarks where the ground-truth treatment effect is known:

  • ACIC 2016: A dataset with 82 covariates (23 continuous, 59 binary) and 4,802 subjects, featuring high treatment heterogeneity.
  • News: A high-dimensional dataset with 3,477 covariates representing word counts in news articles. This represents a significant challenge for traditional matching.
  • IHDP: A smaller dataset (25 covariates, 747 subjects) used to test the method in lower-dimensional settings.

Key Insights and Ideas

  • The Hierarchy of Coarseness: Neural networks naturally provide a spectrum of balancing scores. The raw covariates are the "finest" (most information), and the scalar propensity score is the "coarsest" (least information). Neural score matching allows researchers to pick a level of coarseness that optimizes the bias-variance trade-off.
  • Theoretical Bounds: The paper provides novel proofs showing that reducing imbalance in the latent neural space directly controls imbalance in the original covariate space. They extend this to "non-balancing scores" (where the model might be slightly misspecified), providing a safety net for real-world applications.
  • Multivariate Advantage: In very high dimensions (like the News dataset), matching on a 5D or 10D hidden layer significantly outperforms matching on a 1D propensity score. The extra dimensions act as a buffer, preventing the "random matching" effect that can plague scalar PSM.

Key Results

The experiments demonstrate that Neural Score Matching is highly competitive, particularly in high-dimensional settings:

  • ACIC 2016: The "NN Layer 1" approach (matching on the first hidden layer) consistently outperformed standard Propensity Score Matching (LogReg PS) and raw covariate matching in terms of ATT (Average Treatment Effect on the Treated) error and sample imbalance.
  • News Dataset: In this high-dimensional regime, multivariate scores (NN Layer 1 and PCA) were the only methods that performed reliably. Scalar propensity scores (both neural and logistic) led to high errors, likely because they discarded too much covariate information.
  • Calibration: The neural network models achieved lower calibration error (better estimation of the true probability of treatment) than standard logistic regression.
Table 1: Results on the ACIC2016 dataset.

Limitations and Future Work

  • Architecture Sensitivity: There is a trade-off between model capacity (wide layers for better propensity estimation) and matching efficiency (narrow layers for better matching). The authors noted that hyperparameters didn't always perform consistently across different datasets.
  • Model Specification: The theoretical guarantees rely on the propensity score model being correctly specified. While Proposition 7 addresses violations, real-world performance is still tied to the quality of the neural network's fit.
  • Future Directions: The authors suggest using their theoretical bounds directly in the loss function to "learn" the best matching space, and exploring the use of multiple intermediate layers simultaneously to further refine the balance.

Summary updated

Abstract

Traditional methods for matching in causal inference are impractical for high-dimensional datasets. They suffer from the curse of dimensionality: exact matching and coarsened exact matching find exponentially fewer matches as the input dimension grows, and propensity score matching may match highly unrelated units together. To overcome this problem, we develop theoretical results which motivate the use of neural networks to obtain non-trivial, multivariate balancing scores of a chosen level of coarseness, in contrast to the classical, scalar propensity score. We leverage these balancing scores to perform matching for high-dimensional causal inference and call this procedure neural score matching. We show that our method is competitive against other matching approaches on semi-synthetic high-dimensional datasets, both in terms of treatment effect estimation and reducing imbalance.
BibTeX
@inproceedings{clivio2022neural,
  author = {Oscar Clivio and Fabian Falck and Brieuc Lehmann and George Deligiannidis and Chris Holmes},
  title = {{Neural Score Matching for High-Dimensional Causal Inference}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=z5v41EmFWVI},
}
Figure