TL;DR
This paper introduces Scalable Random Feature Latent Variable Models (SRFLVMs), a new framework that makes powerful RFLVMs practical for large datasets. By combining a novel stick-breaking construction for Dirichlet Processes with a block coordinate descent variational inference (BCD-VI) algorithm, SRFLVMs achieve significant speedups and improved performance in learning latent representations and handling missing data, outperforming existing methods.
What Problem Does This Paper Address?
Latent Variable Models (LVMs) are excellent for uncovering hidden structures in complex, high-dimensional data. Random Feature Latent Variable Models (RFLVMs) are a particularly powerful class of LVMs, known for their ability to handle non-Gaussian data and learn rich representations. However, RFLVMs traditionally rely on computationally expensive Monte Carlo (MC) sampling methods for inference. This reliance makes them impractically slow for large datasets, a major hurdle in the era of "big data." The paper aims to bridge this gap by developing a scalable inference method for RFLVMs.
Methods
The core of the paper's contribution lies in two key innovations:
- Stick-Breaking Construction for Dirichlet Processes (DP): DPs are often used in RFLVMs to provide flexible kernel mixtures. However, standard DP formulations lack explicit probability density functions (PDFs) for their mixing weights, making them difficult to integrate into variational Bayesian inference (VBI). The authors introduce a stick-breaking construction that provides a tractable PDF for these weights, enabling their use within VBI.
- Block Coordinate Descent Variational Inference (BCD-VI): Existing VBI methods often struggle with the high dimensionality of variational parameters in models like RFLVMs. BCD-VI addresses this by partitioning the variational parameters into distinct blocks. It then optimizes these blocks sequentially using specialized solvers: reparameterization gradient variational inference (RGVI) for non-conjugate terms and a more efficient method for conjugate terms. This "divide and conquer" approach significantly improves computational efficiency and scalability.
These two components are combined to create the Scalable RFLVM (SRFLVM) framework.
Datasets
The paper evaluates SRFLVMs on a variety of datasets:
- Synthetic Datasets: A 2-D S-shape manifold dataset was synthesized from a GPLVM to test manifold learning and kernel learning capabilities.
- Real-World Datasets:
- Image Datasets: CIFAR-10 and MNIST were used for classification and missing data imputation tasks. For RFLVM comparisons, these datasets were downsampled due to computational constraints.
- Other Datasets: BRIDGES (bicycle counts), CIFAR-10, MNIST, MONTREAL (bicycle counts), NEWSGROUPS (text data), and YALE (faces) were used for classification tasks.
- Missing Data Imputation: CIFAR and BRENDAN (faces) datasets were used to evaluate imputation performance.
Key Insights and Ideas
- VBI is the Key to Scalability: The paper strongly advocates for Variational Bayesian Inference (VBI) as the path to making complex models like RFLVMs scalable, moving beyond the limitations of traditional MC sampling.
- DPs are Tractable with Stick-Breaking: The stick-breaking construction is a crucial insight, showing how to make flexible DP priors compatible with optimization-based inference methods like VBI.
- Block-wise Optimization is More Efficient: BCD-VI highlights that treating variational parameters as a monolithic block is inefficient. By breaking down the problem and using tailored solvers for each block, significant computational gains can be achieved.
- Likelihood Choice Matters: The experiments show that for certain datasets (e.g., MONTREAL), using a negative binomial likelihood (N-RFLVM) outperforms a Gaussian likelihood (G-RFLVM), underscoring the importance of model adaptability.
- Approximate Inference Can Be as Good: The comparable performance of SRFLVMs (using VBI) and RFLVMs (using MCMC) on manifold learning suggests that approximate inference can achieve similar quality in latent representations.
Key Results
- Scalability: SRFLVMs demonstrate significantly faster wall-times compared to traditional RFLVMs, especially as dataset size (N) and dimensionality (M) increase. For example, Table 3 shows SRFLVMs are often an order of magnitude faster.
- Performance:
- Classification: SRFLVMs (specifically G-OURS and G-RFLVM) achieved the highest KNN classification accuracy on several real-world datasets (Table 2).
- Manifold Learning: SRFLVMs produced latent representations that more closely aligned with ground truth compared to BGPLVM and GPLVM-SVI (Figure 3a).
- Missing Data Imputation: SRFLVMs outperformed other methods in reconstructing missing pixels on CIFAR and BRENDAN datasets, as shown by lower MSE values in Table 5 and visually in Figures 5 and 6.
- Robustness: The method shows good performance across various likelihood functions (Gaussian, Negative Binomial, Bernoulli) and datasets.
Limitations and Future Work
- Limitations:
- The paper notes that for logistic likelihood, approximate inference on stochastic weights (H) can degrade kernel learning quality, motivating the use of Pólya-gamma augmentation.
- While BCD-VI is more efficient, it still involves iterative optimization, and convergence to a global optimum is not always guaranteed (though it is shown to converge to a stationary point).
- Future Work:
- The authors suggest that the BCD-VI framework offers opportunities for further enhancements by integrating advanced VBI algorithms for subproblem solving.
- The paper implicitly suggests exploring more complex likelihoods and kernel structures, as SRFLVMs are designed to be flexible.
Summary updated