Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders

Eleni Sgouritsa, Dominik Janzing, Jonas Peters, Bernhard Schölkopf

Uncovering hidden structures in data is a fundamental challenge in statistics and machine learning. Traditional methods like Gaussian Mixture Models (GMMs) often rely on restrictive assumptions: knowing the number of components beforehand and assuming a specific parametric form, such as a Gaussian distribution. In this paper, Eleni Sgouritsa and colleagues from the Max Planck Institute and ETH Zurich introduce a kernel-based framework that identifies the number of components and clusters data points without these rigid constraints.

The Problem of Latent Confounders

The authors focus on scenarios where observed variables $Y_1, \dots, Y_d$ are independent given a latent variable $W$. This structure is common in causal inference, where $W$ acts as a hidden common cause (confounder) explaining the correlations between observed symptoms or measurements.

Graphical model of a latent variable $W$ influencing observed variables $Y_1, \dots, Y_d$.

Standard modeling approaches fail when the data is nonparametric—meaning it doesn't fit a standard bell curve. Furthermore, in causal discovery, distinguishing whether variable $A$ causes $B$ or if a hidden variable $C$ causes both is notoriously difficult. This paper provides a mathematical toolkit to solve both the "how many groups?" and the "which group does this point belong to?" questions.

Methodology: Rank and Independence

The core of the approach lies in Hilbert Space Embeddings of Distributions. By mapping probability distributions into an infinite-dimensional feature space (a Reproducing Kernel Hilbert Space, or RKHS), the authors prove that the number of mixture components $m$ is equal to the rank of the tensor representing the joint distribution. To make this practical for finite data, they use Singular Value Decomposition (SVD) to estimate this rank.

Following model selection, the authors introduce CLIC (Clustering with Independence Criterion). Unlike distance-based clustering like K-means, CLIC seeks a partition that maximizes the conditional independence of variables within each cluster. It uses the Hilbert Schmidt Independence Criterion (HSIC) as an objective function, iteratively swapping points between clusters to minimize the dependence between variables. If $X$ and $Y$ are independent given $Z$, then once the groups of $Z$ are correctly identified, the correlation between $X$ and $Y$ should vanish within those groups.

Key Results and Causal Discovery

The SVD-based rank estimation proved highly effective in simulations. As shown in the experimental results, the method consistently identifies the correct number of components even for non-Gaussian distributions.

Histogram showing the distribution of the estimated number of components $m$ in a synthetic dataset where the true $m=2$.

In real-world tests, CLIC successfully recovered ground-truth latent variables (such as diagnosis in breast cancer data or sex in arrhythmia data) where standard Expectation-Maximization (EM) methods struggled.

Furthermore, this framework provides a novel tool for causal discovery. By estimating the rank of the joint distribution of two variables, one can distinguish between a hidden common cause (low/finite rank) and a direct causal link (high/infinite rank). On the Tuebingen Cause-Effect Pairs benchmark, the method correctly showed that direct causal links result in much higher estimated ranks than confounded relationships.

Conclusion

This work provides a principled, nonparametric approach to latent variable modeling. By leveraging the geometry of Hilbert space embeddings, the authors move beyond the limitations of parametric models, offering a robust toolkit for discovering hidden structures in complex data. While computationally intensive for very large datasets, the method offers a unique, independence-driven perspective on clustering and causal inference.

Summary updated

Abstract

We propose a kernel method to identify finite mixtures of nonparametric product distributions. It is based on a Hilbert space embedding of the joint distribution. The rank of the constructed tensor is equal to the number of mixture components. We present an algorithm to recover the components by partitioning the data points into clusters such that the variables are jointly conditionally independent given the cluster. This method can be used to identify finite confounders.
BibTeX
@inproceedings{sgouritsa2013identifying,
  author = {Eleni Sgouritsa and Dominik Janzing and Jonas Peters and Bernhard Schölkopf},
  title = {{Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2013},
  url = {https://openreview.net/forum?id=V3UmegHTym},
}
Figure