Abstract: The Indian Buffet Process (IBP) is a nonparametric prior for latent feature models in which observations are influenced by a combination of several hidden features. For example, images may be composed of several objects or sounds may consist of several notes. Latent feature models seek to infer what these latent features from a set of observations. Current inference methods for the IBP have all relied on sampling. While these methods are guaranteed to be accurate in the limit, in practice, samplers for the IBP tend to mix slow. We develop a deterministic variational method for the IBP. We provide theoretical guarantees on its truncation bounds and demonstrate its superior performance for high dimensional data sets.
Introduction
In unsupervised learning, identifying the latent (hidden) features that describe a dataset is a fundamental challenge. The Indian Buffet Process (IBP) provides a powerful nonparametric prior for these models because it allows the number of features to be discovered automatically from the data, rather than requiring a fixed number to be specified in advance.
However, the IBP is computationally demanding. Because it involves an infinite number of potential features and complex combinatorial assignments, researchers have historically relied on Markov Chain Monte Carlo (MCMC) sampling. While theoretically accurate, MCMC methods are often slow to converge (mix), prone to getting stuck in local optima, and scale poorly as the dimensionality of the data increases. This paper introduces the first deterministic inference framework for the IBP, offering a faster and more scalable alternative.
The Latent Feature Model
The IBP is typically used in models where an observation matrix $X$ is decomposed into a binary assignment matrix $Z$ and a feature value matrix $A$. In this context, $z_{nk} = 1$ indicates that observation $n$ possesses feature $k$.
Variational Inference Framework
The authors propose a Variational Inference (VI) approach that treats inference as an optimization problem. Instead of sampling, VI seeks a simpler distribution that closely approximates the complex posterior. Key technical contributions include:
- Stick-Breaking Construction: The method leverages a stick-breaking representation of the IBP to define feature probabilities as a sequence of decreasing weights.
- Truncation: To handle the infinite process, the authors apply a truncation level $K$. They provide theoretical proofs showing that the error introduced by this truncation decreases exponentially as $K$ increases.
- Multinomial Bound: To solve the difficulty of calculating expectations of the log of a product of variables, the authors introduced a multinomial lower bound. Combined with Jensen’s inequality, this allows for efficient, closed-form parameter updates.
- Soft Assignments: Unlike MCMC samplers that use "hard" binary assignments, the variational approach uses "soft" probabilistic assignments. This allows the algorithm to explore the configuration space more fluidly and escape local optima that often trap samplers in high dimensions.
Experimental Results
The method was evaluated on synthetic data, Yale Faces ($D=1032$), and multi-microphone speech recordings. The results consistently showed that the variational approaches were orders of magnitude faster than Gibbs sampling and particle filtering.
On the high-dimensional Yale Faces dataset, the variational methods achieved significantly higher predictive log-likelihoods than the uncollapsed Gibbs sampler. While samplers can be effective for very simple, low-dimensional datasets where they mix quickly, the variational method proved nearly indifferent to dimensionality ($D$), maintaining efficiency where samplers struggled.
Conclusion
This work represents a significant milestone in Bayesian nonparametrics by providing a scalable, deterministic alternative to sampling for the IBP. While the method remains subject to local optima and requires a user-defined truncation level $K$, its superior performance on high-dimensional data makes it a robust tool for latent feature discovery. Future work may explore removing the fixed truncation requirement during optimization.
Summary updated