Factorial Mixture of Gaussians and the Marginal Independence Model

Ricardo Bezerra de Andrade e Silva, Zoubin GhahramaniAISTATS 2009

Introduction

In probabilistic graphical modeling, researchers often focus on conditional independence—the idea that two variables are independent given the knowledge of a third. However, marginal independence, where variables are independent regardless of any other factors, is equally crucial, particularly in sparse systems where data originates from unrelated hidden sources.

Modeling marginal independence in non-Gaussian, non-linear data has historically been challenging. Standard mixture models fail because they use a single global hidden variable to coordinate clusters. This "global switch" inadvertently couples all observed variables, making them dependent even if they are independent within each cluster. This paper, presented at AISTATS 2009, introduces the Factorial Mixture of Gaussians (FMoG) to solve this coupling problem.

The Factorial Mixture of Gaussians (FMoG) Model

The core innovation of FMoG is moving from a single mixture indicator to a vector of "factorial" indicators. Instead of one global variable $c$, every observed variable $Y_i$ is assigned its own indicator $c_i$.

Comparison of standard mixture models (a) and the proposed factorial approach (d).

By factorizing the mixture, the model can capture complex non-linearities without creating spurious dependencies. The parameters are constrained such that the mean and variance of $Y_i$ depend only on its own indicator $c_i$, and the covariance between $Y_i$ and $Y_j$ depends only on the pair $(c_i, c_j)$. If the model's graph dictates that two variables are independent, their covariance is forced to zero.

Graphical Representation and Marginal Independence

The FMoG framework utilizes bi-directed graphs to represent marginal independence structures. In these graphs, the absence of a bi-directed edge between two nodes $Y_i$ and $Y_j$ signifies that they are marginally independent.

A bi-directed graph where the absence of an edge between $Y_1$ and $Y_3$ indicates marginal independence.

This approach is "agnostic" toward latent structures. Unlike Latent Variable Models (LVMs), which require predefined assumptions about the number and connectivity of hidden factors, bi-directed models directly represent observed marginals. This makes them more robust when the true underlying hidden structure is unknown or overly complex.

Learning via Expectation-Maximization

Learning the FMoG parameters involves an Expectation-Maximization (EM) framework. The M-step is particularly complex, requiring non-linear optimization to ensure that covariance matrices remain positive definite while strictly satisfying the independence constraints defined by the graph. The authors utilized the fmincon optimizer in MATLAB to navigate these constraints.

Results and Performance

The researchers evaluated the Bi-directed Model (BDM) against standard Mixtures of Gaussians (MG) and canonical Latent Variable Models (LVM) using several UCI datasets, including Glass, Wine, Statlog Heart, and Forest Fires.

Key findings include:

  • Predictive Power: The BDM consistently matched or outperformed LVM and MG in terms of predictive log-likelihood across five-fold cross-validation.
  • Generalization: On the YEAST dataset, the BDM showed significantly better performance than the LVM, suggesting that enforcing correct marginal independence constraints improves the model's ability to generalize to unseen data.

Limitations and Future Directions

The primary challenge for FMoG is computational scalability. The state space for the indicator vector grows exponentially ($k^p$, where $k$ is the number of mixture components and $p$ is the number of variables), making exact EM intractable for high-dimensional data. Future work will focus on developing specialized approximation methods, as standard "mean-field" techniques are difficult to apply to the dense dependencies found in bi-directed graphs. Additionally, the authors envision FMoG as a building block for "mixed graph" models that combine both directed (causal) and bi-directed edges.

Summary updated

BibTeX
@article{silva2009factorial,
  author = {Ricardo Bezerra de Andrade e Silva and Zoubin Ghahramani},
  title = {{Factorial Mixture of Gaussians and the Marginal Independence Model}},
  journal = {AISTATS},
  year = {2009},
}
Figure