Geometry-Aware Instrumental Variable Regression

Heiner Kremer, Bernhard Schölkopf

Introduction

In causal inference, determining the effect of a treatment ($T$) on an outcome ($Y$) is frequently complicated by unobserved confounders. Instrumental Variable (IV) regression addresses this by utilizing an instrument ($Z$) that influences the treatment but has no direct effect on the outcome. Modern IV approaches typically rely on Conditional Moment Restrictions (CMR), where a model is trained to satisfy the condition $E[Y - f(T) | Z] = 0$.

However, most existing CMR-based estimators, such as Generalized Empirical Likelihood (GEL), operate under the assumption that the true data distribution is a reweighted version of the observed samples. This "reweighting" approach is fundamentally limited: it is "blind" to the geometric structure of the data space. If the data is corrupted or subject to small adversarial perturbations, merely adjusting the weights of existing points is insufficient for robust estimation.

Comparison of distribution matching strategies: Divergence-based reweighting ($D_\phi$), Maximum Mean Discrepancy (MMD), and the proposed Sinkhorn distance ($W_c^\epsilon$). Note how the Sinkhorn distance accounts for local movement of mass.

The Sinkhorn Method of Moments (SMM)

To overcome these limitations, researchers have introduced the Sinkhorn Method of Moments (SMM). Instead of treating data points as static entities to be reweighted, SMM employs Optimal Transport (OT) to allow the "mass" of data points to move within the feature space. Specifically, it uses the entropy-regularized Wasserstein distance, known as the Sinkhorn distance, to measure the discrepancy between the empirical distribution and the set of distributions satisfying the causal requirements.

The SMM framework seeks to find a distribution $P$ within the set of distributions satisfying the conditional moment restriction that is closest to the empirical distribution $\hat{P}_n$ according to a geometry-aware distance $R(f)$.

Geometry Awareness and Robustness

The core innovation of SMM lies in its "geometry awareness." Through a mathematical derivation (Taylor expansion) of the dual objective for small regularization values ($\epsilon$), the authors demonstrate that the objective function naturally incorporates the gradient and the Laplacian of the moment function.

This property has profound implications for robustness:

  • Gradient Regularization: Forces the model to be less sensitive to small shifts in data position.
  • Laplacian Regularization: Acts as a "shield" by promoting flatness in the error surface. In an adversarial context, a flat error surface ensures that small, malicious perturbations (like those from a Fast Gradient Sign Method attack) do not lead to large changes in the model's output.

Experimental Results

The SMM approach was evaluated across several benchmarks, demonstrating its superiority in challenging environments:

  • Robustness to Corruption: In experiments where the treatment variable was intentionally "poisoned" with noise, SMM maintained a significantly lower Mean Squared Error (MSE) compared to traditional baselines like Maximum Moment Restriction (MMR).
  • Adversarial Defense: SMM showed remarkable resilience against FGSM (Fast Gradient Sign Method) attacks. While other state-of-the-art methods saw performance collapse as attack strength increased, SMM's error rates remained stable.
  • Standard Performance: On clean datasets (e.g., the NetworkIV benchmark), SMM performed on par with existing non-linear IV estimators, indicating that its robustness does not come at the cost of accuracy in ideal settings.

Conclusion

The Sinkhorn Method of Moments represents a significant step forward in making causal inference tools more reliable for real-world applications. By moving from "geometry-blind" reweighting to "geometry-aware" optimal transport, SMM provides a robust framework capable of handling corrupted data and adversarial threats, bridging the gap between theoretical causal models and the messy reality of modern data science.

Summary updated

Abstract

Instrumental variable (IV) regression can be approached through its formulation in terms of conditional moment restrictions (CMR). Building on variants of the generalized method of moments, most CMR estimators are implicitly based on approximating the population data distribution via reweightings of the empirical sample. While for large sample sizes, in the independent identically distributed (IID) setting, reweightings can provide sufficient flexibility, they might fail to capture the relevant information in presence of corrupted data or data prone to adversarial attacks. To address these shortcomings, we propose the Sinkhorn Method of Moments, an optimal transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings but improves robustness against data corruption and adversarial attacks.
BibTeX
@inproceedings{kremer2024geometry,
  author = {Heiner Kremer and Bernhard Schölkopf},
  title = {{Geometry-Aware Instrumental Variable Regression}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=6KLNiRdWH6},
}
Figure