Scalable Out-of-Distribution Robustness in the Presence of Unobserved Confounders

Parjanya Prajakta Prashant, Seyedeh Baharan Khatami, Bruno Ribeiro, Babak Salimi

Abstract

We consider the task of out-of-distribution (OOD) generalization, where the distribution shift is due to an unobserved confounder ($Z$) affecting both the covariates ($X$) and the labels ($Y$). This confounding introduces heterogeneity in the predictor, i.e., $P(Y \mid X) = E_{P(Z \mid X)}[P(Y \mid X,Z)]$, making traditional covariate and label shift assumptions unsuitable. OOD generalization differs from traditional domain adaptation in that it does not assume access to the covariate distribution ($X^\text{te}$) of the test samples during training. These conditions create a challenging scenario for OOD robustness: (a) $Z^\text{tr}$ is an unobserved confounder during training, (b) $P^\text{te}(Z) \neq P^\text{tr}(Z)$, (c) $X^\text{te}$ is unavailable during training, and (d) the predictive distribution depends on $P^\text{te}(Z)$. While prior work has developed complex predictors requiring multiple additional variables for identifiability of the latent distribution, we explore a set of identifiability assumptions that yield a surprisingly simple predictor using only a single additional variable. Our approach demonstrates superior empirical performance on several benchmark tasks.

Introduction: The Challenge of Unobserved Confounders

Most machine learning models assume that the relationship between features ($X$) and labels ($Y$) remains constant, even if the distribution of $X$ changes. However, in real-world scenarios, "unobserved confounders" ($Z$) often influence both features and outcomes. For example, in healthcare, socioeconomic status might affect both patient symptoms and clinical outcomes.

Causal graph illustrating the relationship between unobserved confounder Z, features X, labels Y, and a proxy variable S.

If the distribution of this hidden variable $Z$ shifts between training and testing, the optimal predictor $P(Y|X)$ changes. Traditional domain adaptation fails here because it typically requires access to test-set features during training. Existing proxy methods often rely on multiple auxiliary variables or non-scalable generative models. This work introduces a simpler, more scalable way to handle these latent shifts using minimal extra information, such as a single noisy proxy or data from multiple sources.

Methodology

The proposed framework consists of a two-stage approach designed for theoretical grounding and computational efficiency:

Stage 1: Learning the Latent Distribution

The model uses an encoder-decoder architecture where the encoder $\phi(X)$ estimates the posterior probability of the hidden confounder, $P(Z|X)$. The decoder then attempts to reconstruct a proxy variable $S$ or a source identifier $U$. To prevent degenerate solutions, a novel variance regularizer ($L_{var}$) is introduced, forcing the model to find a representation of $Z$ that is highly certain in specific regions of the feature space.

Stage 2: Mixture-of-Experts (MoE)

With $P(Z|X)$ learned, a Mixture-of-Experts model is trained. Each "expert" specializes in a specific value of the hidden confounder $Z$. The gating mechanism is the $P(Z|X)$ posterior learned in the first stage.

Inference and Shift Estimation

At test time, the proxy variable is no longer needed. Instead, the framework employs Black Box Shift Estimation (BBSE) to detect shifts in the marginal distribution of $Z$ and reweights the MoE gating function accordingly to maintain robustness.

The inference pipeline showing the estimation of weights w via BBSE and the final prediction using the reweighted Mixture-of-Experts model.

Key Insights and Ideas

  • The Blessing of Dimensionality: A key theoretical insight is that as the number of features ($X$) increases, the hidden confounder $Z$ becomes easier to identify. In high-dimensional spaces, features are likely to be sufficiently discriminative to pinpoint the latent state with near-zero error.
  • Minimalist Requirements: Unlike prior work requiring multiple proxies, this method proves that a single proxy or multiple data sources are sufficient for identifiability.
  • Scalability: By avoiding complex generative models like VAEs or GANs, the method achieves linear time complexity relative to sample size, significantly outperforming competitors like ProxyDA which scale cubically.

Experimental Results

The researchers evaluated their approach on synthetic and real-world benchmarks (ACS/Folktables):

  • Synthetic Performance: The method achieved ~89% accuracy, nearly doubling the performance of standard Empirical Risk Minimization (ERM) and beating robust methods like IRM and VREx (~46-52%).
  • Real-World Accuracy: On the ACS Income dataset, the model reached 0.883 accuracy, outperforming ProxyDA (0.856). Similar dominance was observed across tasks like employment prediction and public coverage.
  • Efficiency: Scalability tests showed constant memory usage and linear runtime as sample sizes grew, whereas competitors faced exponential increases.

Conclusion and Limitations

While the framework provides a scalable solution for OOD robustness, it currently assumes the hidden confounder $Z$ is a discrete variable. Extending this to continuous latent variables remains an area for future research. Additionally, performance depends on the proxy having a "full rank" relationship with the hidden confounder to ensure it contains enough information to distinguish between different values of $Z$.

Summary updated

Abstract

We consider the task of out-of-distribution (OOD) generalization, where the distribution shift is due to an unobserved confounder ($Z$) affecting both the covariates ($X$) and the labels ($Y$). This confounding introduces heterogeneity in the predictor, i.e., $P(Y \mid X) = E_{P(Z \mid X)}[P(Y \mid X,Z)]$, making traditional covariate and label shift assumptions unsuitable. OOD generalization differs from traditional domain adaptation in that it does not assume access to the covariate distribution ($X^\text{te}$) of the test samples during training. These conditions create a challenging scenario for OOD robustness: (a) $Z^\text{tr}$ is an unobserved confounder during training, (b) $P^\text{te}(Z) \neq P^\text{tr}(Z)$, (c) $X^\text{te}$ is unavailable during training, and (d) the predictive distribution depends on $P^\text{te}(Z)$. While prior work has developed complex predictors requiring multiple additional variables for identifiability of the latent distribution, we explore a set of identifiability assumptions that yield a surprisingly simple predictor using only a single additional variable. Our approach demonstrates superior empirical performance on several benchmark tasks.
BibTeX
@inproceedings{prashant2025scalable,
  author = {Parjanya Prajakta Prashant and Seyedeh Baharan Khatami and Bruno Ribeiro and Babak Salimi},
  title = {{Scalable Out-of-Distribution Robustness in the Presence of Unobserved Confounders}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=eIyOtZ9tgl},
}
Figure