Scalable Random Feature Latent Variable Models

Ying Li, Zhidi Lin, Yuhao Liu, Michael Minyi Zhang, Pablo M. Olmos, Petar M. DjuricIEEE TPAMI 2024 · OpenReview

TL;DR

This paper introduces Scalable Random Feature Latent Variable Models (SRFLVMs), a new framework that makes powerful RFLVMs practical for large datasets. By combining a novel stick-breaking construction for Dirichlet Processes with a block coordinate descent variational inference (BCD-VI) algorithm, SRFLVMs achieve significant speedups and improved performance in learning latent representations and handling missing data, outperforming existing methods.

What Problem Does This Paper Address?

Latent Variable Models (LVMs) are excellent for uncovering hidden structures in complex, high-dimensional data. Random Feature Latent Variable Models (RFLVMs) are a particularly powerful class of LVMs, known for their ability to handle non-Gaussian data and learn rich representations. However, RFLVMs traditionally rely on computationally expensive Monte Carlo (MC) sampling methods for inference. This reliance makes them impractically slow for large datasets, a major hurdle in the era of "big data." The paper aims to bridge this gap by developing a scalable inference method for RFLVMs.

Methods

The core of the paper's contribution lies in two key innovations:

  • Stick-Breaking Construction for Dirichlet Processes (DP): DPs are often used in RFLVMs to provide flexible kernel mixtures. However, standard DP formulations lack explicit probability density functions (PDFs) for their mixing weights, making them difficult to integrate into variational Bayesian inference (VBI). The authors introduce a stick-breaking construction that provides a tractable PDF for these weights, enabling their use within VBI.
  • Block Coordinate Descent Variational Inference (BCD-VI): Existing VBI methods often struggle with the high dimensionality of variational parameters in models like RFLVMs. BCD-VI addresses this by partitioning the variational parameters into distinct blocks. It then optimizes these blocks sequentially using specialized solvers: reparameterization gradient variational inference (RGVI) for non-conjugate terms and a more efficient method for conjugate terms. This "divide and conquer" approach significantly improves computational efficiency and scalability.

These two components are combined to create the Scalable RFLVM (SRFLVM) framework.

Datasets

The paper evaluates SRFLVMs on a variety of datasets:

  • Synthetic Datasets: A 2-D S-shape manifold dataset was synthesized from a GPLVM to test manifold learning and kernel learning capabilities.
  • Real-World Datasets:
    • Image Datasets: CIFAR-10 and MNIST were used for classification and missing data imputation tasks. For RFLVM comparisons, these datasets were downsampled due to computational constraints.
  • Other Datasets: BRIDGES (bicycle counts), CIFAR-10, MNIST, MONTREAL (bicycle counts), NEWSGROUPS (text data), and YALE (faces) were used for classification tasks.
  • Missing Data Imputation: CIFAR and BRENDAN (faces) datasets were used to evaluate imputation performance.

Key Insights and Ideas

  • VBI is the Key to Scalability: The paper strongly advocates for Variational Bayesian Inference (VBI) as the path to making complex models like RFLVMs scalable, moving beyond the limitations of traditional MC sampling.
  • DPs are Tractable with Stick-Breaking: The stick-breaking construction is a crucial insight, showing how to make flexible DP priors compatible with optimization-based inference methods like VBI.
  • Block-wise Optimization is More Efficient: BCD-VI highlights that treating variational parameters as a monolithic block is inefficient. By breaking down the problem and using tailored solvers for each block, significant computational gains can be achieved.
  • Likelihood Choice Matters: The experiments show that for certain datasets (e.g., MONTREAL), using a negative binomial likelihood (N-RFLVM) outperforms a Gaussian likelihood (G-RFLVM), underscoring the importance of model adaptability.
  • Approximate Inference Can Be as Good: The comparable performance of SRFLVMs (using VBI) and RFLVMs (using MCMC) on manifold learning suggests that approximate inference can achieve similar quality in latent representations.

Key Results

  • Scalability: SRFLVMs demonstrate significantly faster wall-times compared to traditional RFLVMs, especially as dataset size (N) and dimensionality (M) increase. For example, Table 3 shows SRFLVMs are often an order of magnitude faster.
  • Performance:
    • Classification: SRFLVMs (specifically G-OURS and G-RFLVM) achieved the highest KNN classification accuracy on several real-world datasets (Table 2).
  • Manifold Learning: SRFLVMs produced latent representations that more closely aligned with ground truth compared to BGPLVM and GPLVM-SVI (Figure 3a).
  • Missing Data Imputation: SRFLVMs outperformed other methods in reconstructing missing pixels on CIFAR and BRENDAN datasets, as shown by lower MSE values in Table 5 and visually in Figures 5 and 6.
  • Robustness: The method shows good performance across various likelihood functions (Gaussian, Negative Binomial, Bernoulli) and datasets.

Limitations and Future Work

  • Limitations:
    • The paper notes that for logistic likelihood, approximate inference on stochastic weights (H) can degrade kernel learning quality, motivating the use of Pólya-gamma augmentation.
  • While BCD-VI is more efficient, it still involves iterative optimization, and convergence to a global optimum is not always guaranteed (though it is shown to converge to a stationary point).
  • Future Work:
    • The authors suggest that the BCD-VI framework offers opportunities for further enhancements by integrating advanced VBI algorithms for subproblem solving.
  • The paper implicitly suggests exploring more complex likelihoods and kernel structures, as SRFLVMs are designed to be flexible.

Summary updated

Abstract

Random feature latent variable models (RFLVMs) represent the state-of-the-art in latent variable models, capable of handling non-Gaussian likelihoods and effectively uncovering patterns in high-dimensional data. However, their heavy reliance on Monte Carlo sampling results in scalability issues which makes it difficult to use these models for datasets with a massive number of observations. To scale up RFLVMs, we turn to the optimization-based variational Bayesian inference (VBI) algorithm which is known for its scalability compared to sampling-based methods. However, implementing VBI for RFLVMs poses challenges, such as the lack of explicit probability distribution functions (PDFs) for the Dirichlet process (DP) in the kernel learning component, and the incompatibility of existing VBI algorithms with RFLVMs. To address these issues, we introduce a stick-breaking construction for DP to obtain an explicit PDF and a novel VBI algorithm called ``block coordinate descent variational inference" (BCD-VI). This enables the development of a scalable version of RFLVMs, or in short, SRFLVM. Our proposed method shows scalability, computational efficiency, superior performance in generating informative latent representations and the ability of imputing missing data across various real-world datasets, outperforming state-of-the-art competitors.
BibTeX
@article{li2024scalable,
  author = {Ying Li and Zhidi Lin and Yuhao Liu and Michael Minyi Zhang and Pablo M. Olmos and Petar M. Djuric},
  title = {{Scalable Random Feature Latent Variable Models}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=Kkh8RnhWga},
}
Figure