Generalization and Invariances in the Presence of Unobserved Confounding

Alexis Bellot, Mihaela van der Schaar

Abstract

Standard domain generalization (DG) methods often fail when unobserved variables—hidden confounders—influence both features and targets. This paper introduces Derivative Invariant Risk Minimization (DIRM), a novel objective that achieves robustness by encouraging the consistency of loss function derivatives across training environments. DIRM bridges the gap between correlation-based models and pure causal solutions, providing a practical framework for generalization in complex, real-world settings like healthcare.

Introduction

Machine learning models frequently exploit "spurious correlations"—patterns present in training data that do not persist in deployment. This is the central challenge of Domain Generalization (DG): training a model on limited environments such that it performs reliably on unseen ones.

Existing invariant methods, such as Invariant Risk Minimization (IRM), typically assume all relevant variables are observed. However, real-world datasets are often plagued by unobserved confounders. For instance, in clinical imaging, a hospital’s specific protocol might influence both the visual appearance of an X-ray and the likelihood of a diagnosis. When these confounders remain hidden, standard causal invariance is violated, leading models to either overfit to environment-specific noise or become too conservative.

Methodology: Derivative Invariant Risk Minimization (DIRM)

The authors propose DIRM to address the limitations of existing DG methods in confounded settings.

The Core Insight

While absolute loss may fluctuate across environments due to hidden confounders, the gradient (derivative) of the loss with respect to model parameters should remain stable if the model captures the underlying causal mechanism. DIRM adds a regularization term to the standard Empirical Risk Minimization (ERM) loss that penalizes the variance of these gradients across training environments:

$$\text{minimize } \mathbb{E}[L] + \lambda \cdot \text{Var}_{e \in \mathcal{E}} (\mathbb{E}_{P_e} [\nabla_\beta L])$$

Robustness and Interpolation

DIRM is mathematically linked to Distributionally Robust Optimization (DRO). It upper-bounds a robust optimization problem over affine combinations of training distributions, enabling extrapolation to unseen environments. By tuning the hyperparameter $\lambda$, DIRM can interpolate between a standard OLS/ERM solution ($\lambda=0$) and a strictly invariant causal solution ($\lambda \to \infty$).

Figure 1: Performance of DIRM vs. baselines under varying levels of unobserved confounding. The model effectively interpolates between OLS and causal solutions as lambda increases.

Experimental Evaluation

The authors evaluated DIRM across synthetic benchmarks, healthcare datasets, and standard computer vision tasks.

  • Pneumonia Diagnosis: Utilizing chest X-rays from the NIH and GMC, where site-specific artifacts act as spurious correlations.
  • Parkinson’s Disease: Using the UCI Parkinson Speech Dataset with varying recording conditions.
  • Survival Prediction: Analyzing MAGGIC heart failure records with "Age" intentionally hidden to serve as an unobserved confounder.
  • DomainBed Benchmarks: Standard DG datasets including VLCS, PACS, and Office-Home.
Figure 6: Sample chest X-rays from GMC and NIH datasets illustrating site-specific visual artifacts that can lead to spurious correlations.

Key Results and Insights

DIRM demonstrates superior performance and stability compared to traditional methods:

  • Predictive Accuracy: In the Pneumonia task, DIRM achieved 63.1% accuracy, significantly higher than ERM (52.7%) and IRM (58.6%). Similar gains were observed in Parkinson's diagnosis (72.4% vs. 62.8% for ERM) and survival prediction (77.6%).
  • Handling Irreducible Error: Unlike methods that penalize variance in the loss itself (e.g., REx), DIRM is robust to changes in irreducible error across environments. This makes it particularly effective for medical data where noise levels vary by site.
  • Feature Consistency: In the MAGGIC dataset, DIRM identified 6 out of the top 10 influential features consistently across 100 environment pairs, compared to only 4 for ERM, highlighting its value for clinical interpretability.
  • General Performance: On standard DomainBed benchmarks, DIRM remained competitive with state-of-the-art algorithms, indicating that its specialized robustness does not come at the cost of general performance.

Conclusion

DIRM offers a robust alternative to standard DG methods by focusing on gradient invariance rather than loss invariance. This approach is uniquely suited for real-world scenarios involving unobserved confounding. Future work will address optimization sensitivities related to the $\lambda$ annealing schedule and explore more complex distribution shifts to further enhance the model's extrapolation capabilities.

Summary updated

BibTeX
@misc{bellot2020generalization,
  author = {Alexis Bellot and Mihaela van der Schaar},
  title = {{Generalization and Invariances in the Presence of Unobserved Confounding}},
  year = {2020},
  eprint = {2007.10653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.10653},
}
Figure