Abstract
Latent variable models represent hidden structure in observational data. To account for the distribution of the observational data changing over time, space, or some other covariate, we need generalizations of these models that allow for dependency. This paper introduces the Dependent Indian Buffet Process (dIBP), a nonparametric Bayesian model for latent features that evolve across a continuous covariate.
Introduction
Standard latent variable models, such as Factor Analysis or the basic Indian Buffet Process (IBP), typically assume that data points are independent and identically distributed (i.i.d.) or exchangeable. However, in many real-world applications, data is indexed by a covariate like time or geography, and the underlying features are expected to vary smoothly. For instance, the spectral features of music change over time, and national development indicators shift with economic growth. While Dependent Dirichlet Processes (DDP) addressed this for mixture models, a corresponding framework for latent feature models—where items can possess multiple features simultaneously—was previously lacking for arbitrary continuous covariates.
Methodology: The Dependent Indian Buffet Process
The dIBP extends the IBP by introducing dependency through a Gaussian Process (GP) thresholding mechanism. The IBP defines a distribution over binary matrices with an infinite number of columns, where each column represents a feature. To make these features dependent on a covariate $t$, the authors employ a "thresholding trick":
- Gaussian Process Coupling: Instead of sampling binary indicators directly, the model draws continuous latent functions $f_k(t)$ from a GP for each feature $k$.
- Thresholding: A feature $k$ is considered active for an observation at $t$ if $f_k(t)$ exceeds a specific threshold. The smoothness of the GP ensures that the probability of feature activation changes continuously across the covariate space.
- Hierarchical Modeling: A hierarchical GP structure is proposed to distinguish between a global feature profile and individual variations among items sharing that feature.
- Inference: Posterior inference is performed using a Markov Chain Monte Carlo (MCMC) approach, specifically a truncated Gibbs sampler for the infinite feature space and Hybrid Monte Carlo (HMC) for learning GP hyperparameters.
Experimental Evaluation
The dIBP was evaluated across synthetic and real-world datasets to test its ability to recover dynamic features and improve predictive performance.
Synthetic Data and Feature Recovery
In synthetic experiments involving 8x8 image features evolving over 40 time steps, the dIBP successfully recovered both the spatial structure of the features and their temporal activation curves. These curves represent the probability of a feature being present at any given point along the covariate.
Piano Music and UN Development Data
On a dataset of piano music spectral features, the dIBP achieved a Root Mean Squared Error (RMSE) of 0.69, significantly outperforming the standard IBP (RMSE 1.08), demonstrating the importance of modeling temporal correlations. Similarly, in an analysis of UN development indicators indexed by GDP, the dIBP (RMSE 0.79) outperformed the static IBP (RMSE 0.93), confirming that development "features" are strongly tied to economic output.
Discussion and Conclusion
The dIBP provides a flexible framework for modeling evolving latent features by bridging discrete feature indicators with continuous Gaussian Processes. This allows researchers to utilize various GP kernels to define complex dependency structures. While the current MCMC-based inference is computationally intensive, future work could explore more scalable techniques like Variational Inference. The dIBP represents a significant step forward in nonparametric Bayesian modeling for non-exchangeable data.
Summary updated