EM Optimization of Latent-Variable Density Models

NeurIPS 1995

TL;DR

This paper introduces a novel algorithm for training latent-variable density models using the Expectation-Maximization (EM) algorithm. This approach allows for the discovery of underlying "causes" that explain complex data, offering a more powerful alternative to traditional linear models, especially for continuous data.

What Problem Does This Paper Address?

Traditional density estimation methods, like mixture models, often rely on linear combinations of basis functions. While effective for some tasks, they struggle to uncover intricate structures in data where a few underlying "causes" interact to produce apparent complexity. Discovering these hidden "causes" is crucial for understanding complex datasets. Previous attempts at building non-linear latent-variable models, while powerful, often required computationally intensive training procedures. This paper aims to bridge this gap by developing a more efficient training method for these models.

Methods

The authors propose a latent-variable density model that uses a non-linear mapping from a lower-dimensional latent space (denoted by $x$) to the observed data space (denoted by $t$). The core of their contribution lies in developing a tractable Expectation-Maximization (EM) algorithm for training this model. Key steps include:

  • Generalized Linear Network for Mapping: The non-linear mapping $y(x; W)$ is modeled using a generalized linear network, which uses a set of fixed basis functions $\phi_j(x)$ and a weight matrix $W$. This offers universal approximation capabilities and is suitable for small latent dimensions.
  • Monte Carlo Approximation: Integrals over the latent space, which are generally intractable, are approximated using Monte Carlo sampling.
  • Shared Samples: A crucial optimization is using the same set of latent samples for all data points within an iteration, simplifying the objective function.
  • EM Algorithm: The resulting objective function, a mixture of kernel functions, allows for the application of the EM algorithm. The E-step involves computing responsibilities (posterior probabilities), and the M-step updates the model weights and noise variance.
  • Regularization: The paper also discusses how to incorporate regularization terms for controlling model complexity, leading to a modified weight update equation.

Datasets

The paper evaluates their algorithm on two types of data:

  • Toy Problem: A synthetic dataset generated from a one-dimensional distribution embedded in two dimensions. This is used as a simple demonstration of the algorithm's capabilities.
  • Oil Flow Data: A 12-dimensional dataset representing measurements from dual-energy gamma densitometers in a multi-phase oil pipeline. This dataset consists of 1000 points, equally distributed among three different geometrical configurations (stratified, homogeneous, and annular flows). This is a more realistic benchmark.

Key Insights and Ideas

  • EM for Latent Variable Models: The paper demonstrates that the EM algorithm can be effectively applied to train non-linear latent-variable density models, making them more computationally accessible.
  • Non-linearity for Better Separation: The non-linear mapping in their model allows for significantly better class separation in the oil flow data compared to linear methods like Principal Component Analysis (PCA) and Factor Analysis.
  • Visualization through Latent Space: The latent space provides a powerful visualization tool. Instead of projecting data points to single locations, the posterior distribution in the latent space offers richer information about how data points are generated.
  • Connection to Existing Models: The model is shown to generalize factor analysis (linear mapping) and has similarities to topographic map algorithms and "soft" principal curves.

Key Results

  • Toy Problem: Figure 2 shows that the algorithm can successfully reconstruct a one-dimensional curve embedded in two dimensions, with the EM algorithm converging to a good fit.
  • Oil Flow Data: Figure 3 illustrates that the non-linear latent-variable model achieves much better separation of the three oil flow configurations in the latent space compared to PCA. This improved separation is attributed to the model's ability to capture non-linear relationships.

Limitations and Future Work

  • Basis Function Growth: The paper notes that for generalized linear networks, the number of basis functions can grow exponentially with the latent dimensionality. However, they argue this is not a significant issue in their context, as the latent dimensionality is typically small (e.g., L=2 for visualization).
  • Missing Data: The authors mention that the model can be extended to handle missing data by integrating out the missing values, leading to a modified EM algorithm.
  • Mixed Variable Types: They are currently extending the model to handle datasets with both continuous and categorical variables.
  • Bayesian Approaches: Future work includes exploring Bayesian approaches using Markov Chain Monte Carlo (MCMC) to replace the maximum likelihood procedure.

Summary updated

BibTeX
@inproceedings{1995em,
  title = {{EM Optimization of Latent-Variable Density Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1995},
}
Figure