Switching Regression Models and Causal Inference in the Presence of Discrete Latent Variables

Rune Christiansen, Jonas Peters

TL;DR

This paper introduces a novel method, ICPH, for causal discovery that can handle unobserved discrete variables (latent variables). It extends the principle of invariant causal prediction by introducing "h-invariance" and provides a robust framework for identifying causal relationships even when some direct causes are hidden. This is crucial for real-world applications where unobserved factors often influence observed data.

What Problem Does This Paper Address?

In many scientific and real-world applications, understanding causal relationships is paramount. Causal discovery aims to infer these relationships from data. A powerful approach, Invariant Causal Prediction (ICP), leverages data from different environments (e.g., different experimental conditions or settings) to identify causal predictors. ICP assumes that the causal mechanism generating a target variable from its direct causes remains invariant across these environments. However, a significant limitation of ICP is its inability to handle situations where some direct causes are unobserved (latent variables). When latent variables are present, the invariance assumption can be violated, leading ICP to fail or produce uninformative results. This paper addresses this gap by developing a method that can perform causal discovery in the presence of discrete latent variables.

Methods

The core of the paper's contribution lies in extending the invariant causal prediction framework to handle discrete latent variables.

  • h-Invariance: The authors introduce the concept of "h-invariance," a relaxed version of the original invariance assumption. This property states that while the direct causal mechanism might change due to latent variables, the relationship between the target variable and its observable direct causes remains invariant across environments, provided the latent variable takes on only a few discrete values.
  • Switching Regression Models: The h-invariance property naturally leads to the use of switching regression models. These models assume that the data generating process for the target variable can be described by one of several regression models, with the choice of model determined by the discrete latent variable.
  • Test for Equality of Switching Regression Models: A key technical contribution is the development of a test to determine if switching regression models are equal across different environments. This test is crucial for verifying the h-invariance property.
  • Invariant Causal Prediction with Hidden Variables (ICPH): The authors propose an algorithm called ICPH that combines the h-invariance concept with the developed statistical tests. ICPH aims to infer the set of observable causal predictors by identifying sets that exhibit h-invariance. The algorithm outputs the intersection of all such h-invariant sets, providing a guarantee of false discovery control.
  • Maximum Likelihood Estimation (MLE) and Confidence Regions: The paper also details the theoretical underpinnings for estimating parameters in switching regression models, including conditions for the existence, consistency, and asymptotic normality of the MLE. It also constructs adjusted confidence regions to address the label-switching problem inherent in mixture models.

Datasets

The paper uses simulated data for its experiments.

  • Simulated Data: The authors generate data from a Structural Causal Model (SCM) that includes discrete latent variables and environments. They vary parameters such as sample size, the number of latent states, and the degree of model misspecification to test their method's performance.
  • Real-World Data: The method is also applied to a real-world dataset related to sun-induced fluorescence (SIF) and land cover classification. This dataset involves incident shortwave radiation (SW), photosynthetically active radiation (APARchl), and gross primary productivity (GPP) as predictors for SIF. The vegetation type is treated as a latent variable.

Key Insights and Ideas

  • Latent variables are a major hurdle for causal discovery: The paper clearly demonstrates why standard invariant causal prediction fails when latent variables are present, highlighting the need for new approaches.
  • "h-invariance" is a powerful relaxation: By introducing h-invariance, the authors provide a principled way to adapt the invariance principle to settings with unobserved discrete factors. This concept is central to their method.
  • Switching regression models are the natural consequence: The mathematical formulation of h-invariance directly leads to the use of switching regression models, making the connection between latent variables and model structure explicit.
  • A robust framework for causal discovery with hidden variables: ICPH offers a theoretically grounded and empirically validated method for identifying causal relationships in complex scenarios where unobserved factors are common.
  • False discovery control is maintained: Even when the underlying assumptions are slightly violated (e.g., in small sample sizes), the proposed method demonstrates good control over false discoveries, making it reliable in practice.

Key Results

  • Theoretical Guarantees: The paper provides theoretical guarantees for the consistency and asymptotic normality of the MLE in switching regression models. It also establishes asymptotic level guarantees for the test of switching regression model equality and asymptotic false discovery control for the ICPH method.
  • Empirical Performance:
    • On simulated data, ICPH successfully identifies causal predictors even in the presence of latent variables.
  • The method shows good performance across various sample sizes and model violations, maintaining type I error control.
  • The power of the method decreases with an increasing number of latent states, as expected.
  • On the real-world SIF dataset, ICPH correctly identifies APARchl as a causal predictor of SIF, and the vegetation type (treated as latent) can be reconstructed with high accuracy.

Limitations and Future Work

  • Number of Latent States: ICPH requires the number of latent states to be specified as an input. While the paper proposes a modification to handle unknown numbers of states (by testing for different degrees), the power of the method decreases as the number of states increases. The authors suggest limiting the application to cases where the number of hidden variables is expected to be small.
  • Identifiability: The paper notes that identifiability of causal predictors cannot be generally achieved if the hidden variable may take arbitrarily many states.
  • Model Violations: While the method shows robustness to several model violations, the assumption on the state space of H (discrete) is crucial for power. For continuous hidden variables, the method often outputs an empty set, which is correct but uninformative.
  • Future Work:
    • Exploring relaxations of the h-invariance assumption.
  • Extending theoretical results to scenarios with dependence between X and H.
  • Developing methods for non-linear models and constructing conditional independence tests for mixture model structures.
  • Investigating the performance with an increasing number of hidden states and predictors.

Summary updated

Abstract

Given a response $Y$ and a vector $X = (X^1, \dots, X^d)$ of $d$ predictors, we investigate the problem of inferring direct causes of $Y$ among the vector $X$. Models for $Y$ that use all of its causal covariates as predictors enjoy the property of being invariant across different environments or interventional settings. Given data from such environments, this property has been exploited for causal discovery. Here, we extend this inference principle to situations in which some (discrete-valued) direct causes of $ Y $ are unobserved. Such cases naturally give rise to switching regression models. We provide sufficient conditions for the existence, consistency and asymptotic normality of the MLE in linear switching regression models with Gaussian noise, and construct a test for the equality of such models. These results allow us to prove that the proposed causal discovery method obtains asymptotic false discovery control under mild conditions. We provide an algorithm, make available code, and test our method on simulated data. It is robust against model violations and outperforms state-of-the-art approaches. We further apply our method to a real data set, where we show that it does not only output causal predictors, but also a process-based clustering of data points, which could be of additional interest to practitioners.
BibTeX
@article{christiansen2020switching,
  author = {Rune Christiansen and Jonas Peters},
  title = {{Switching Regression Models and Causal Inference in the Presence of Discrete Latent Variables}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
}
Figure