Robust Estimation of Causal Heteroscedastic Noise Models

Quang-Duy Tran, Bao Duong, Phuoc Nguyen, Thin Nguyen

Identifying cause-and-effect relationships from observational data—causal discovery—is a cornerstone of scientific inquiry. While Structural Causal Models (SCMs) provide a framework for this, real-world data often exhibits heteroscedasticity, where the variance of the noise depends on the cause. This paper introduces ROCHE (Robust estimation approach for Causal HEteroscedastic noise models), a framework that moves beyond standard Gaussian assumptions to provide more stable and accurate causal inference.

The Problem: The "Gaussian Trap"

Most existing methods for Heteroscedastic Noise Models (HNMs) assume that noise follows a Gaussian (normal) distribution. However, Gaussian models are "light-tailed" and highly sensitive to outliers or regions with low sample density. In these scenarios, the model may produce suboptimal fits that lead to incorrect causal conclusions. For instance, as a cause $X$ increases, the variability of effect $Y$ might grow; a Gaussian model often struggles to capture this transition accurately without being skewed by extreme values.

Comparison of Gaussian vs. Student’s t-distribution fits for heteroscedastic noise. The Student's t-distribution (right) provides a more robust fit and cleaner residuals in the presence of varying variance.

The ROCHE Framework

The core innovation of ROCHE is the replacement of the standard Gaussian likelihood with a Student’s $t$-distribution. Unlike the Gaussian, the Student’s $t$-distribution is "heavy-tailed," controlled by a "degrees of freedom" parameter ($\nu$). This allows the model to account for extreme values without compromising the overall fit.

Key Methodological Components:

  1. Neural Network Parameterization: ROCHE utilizes neural networks to estimate three critical parameters as functions of the cause $X$: the location ($\mu$), the scale ($\tau^2$), and the degrees of freedom ($\nu$).
  2. Stability Constraints: To prevent variance from "exploding" in low-density regions, the authors implement a specific parameterization and a constraint loss function. This encourages the model to remain "Gaussian-like" unless the data specifically necessitates heavy tails.
  3. Independence Testing: After fitting models for both $X \to Y$ and $Y \to X$, ROCHE calculates residuals and employs the Hilbert-Schmidt Independence Criterion (HSIC). The direction that yields residuals most independent of the purported cause is identified as the true causal direction.

Experimental Evaluation and Results

The authors evaluated ROCHE against 13 benchmarks, including 12 synthetic datasets (covering Additive Noise, Location-Scale, and Multiplicative Noise) and the real-world Tübingen cause-effect pairs.

Performance comparison of ROCHE against state-of-the-art methods (LOCI, GRCI, CAM, etc.) across various noise models and metrics.

ROCHE consistently outperformed state-of-the-art baselines like LOCI, GRCI, and CAM:

  • Superior Accuracy: ROCHE achieved an average accuracy of 87.69% across all benchmarks, surpassing the next best method (LOCI at 84.24%).
  • Robustness: The model showed significant gains on the Tübingen dataset, demonstrating its effectiveness on messy, real-world data where Gaussian-based methods often fail.
  • Consistency: Across various noise types (AN, LS, MN-U), ROCHE maintained high Accuracy, AUDRC, and AUROC, indicating its versatility.

Conclusion

ROCHE demonstrates that embracing heavy-tailed distributions like the Student’s $t$ significantly improves the reliability of causal discovery in heteroscedastic environments. By learning to be "Gaussian where possible and robust where necessary," ROCHE avoids the pitfalls of model misspecification. Future work aims to extend this bivariate framework to multivariate systems and complex causal networks.

Summary updated

BibTeX
@inproceedings{tran2024robust,
  author = {Quang-Duy Tran and Bao Duong and Phuoc Nguyen and Thin Nguyen},
  title = {{Robust Estimation of Causal Heteroscedastic Noise Models}},
  booktitle = {SDM},
  year = {2024},
  doi = {10.1137/1.9781611978032.90},
}
Figure