TL;DR
This paper introduces a Restricted Hierarchical Beta Process (R-HBP) and a novel slice sampler that enables exact, tractable inference for hierarchical factor analysis. By restricting the top-level concentration parameter, the authors can use a stick-breaking construction to share an infinite number of latent factors across multiple data sources, even when the model's likelihood and priors are non-conjugate.
What Problem Does This Paper Address?
In Bayesian multi-source modeling, we often want to discover "factors" (latent patterns) that are shared across different datasets. For example, in text analysis, different research fields might share common "topics." The Hierarchical Beta Process (HBP) is a powerful tool for this, as it allows for an infinite library of factors where each data source selects a subset.
However, HBP has a major "inference bottleneck." To make the math work, researchers usually require conjugacy—a mathematical convenience where the prior and likelihood distributions "match." Without it, you can't easily integrate out parameters to find new factors. Previous work-arounds involved arbitrary truncation (manually capping the number of factors) or crude approximations that lead to inaccurate results. This paper aims to provide a way to perform exact inference without these compromises.
Methods
The authors propose two main innovations:
- Restricted Hierarchical Beta Process (R-HBP): They fix the concentration parameter of the parent beta process to one ($\gamma_0 = 1$). This specific restriction allows them to link the HBP to the Indian Buffet Process (IBP) and its "stick-breaking" construction.
- The Slice Sampler: This is the "secret sauce." Instead of dealing with an infinite number of factors, the sampler introduces an auxiliary "slice" variable ($\rho$). In each iteration, this variable effectively "slices" the infinite process, leaving only a finite number of active factors to be computed. This makes the infinite representation tractable without needing to pre-specify a truncation limit.
They apply this framework to two specific models:
- NHFA-PGM: A Poisson-Gamma model designed for count data (like word frequencies in text).
- NHFA-GGM: A Gaussian-Gaussian model for real-valued data (like image features).
Datasets
The authors evaluated their approach on three types of data:
- Synthetic Data: 100-dimensional binary factors used to verify if the model could correctly recover the "ground truth" number of factors and their scores.
- NIPS 0-12 Dataset: A collection of 1,564 neural information processing articles divided into nine categories (e.g., Neuroscience, Learning Theory). This was used to test "transfer learning" by using one category as a target and others as auxiliary sources.
- NUS-WIDE Animal Dataset: 3,411 images across 13 animal categories, using low-level features like color histograms and edge direction.
Key Insights and Ideas
- Exact vs. Approximate: The slice sampler provides "exact" inference in the sense that it samples from the true posterior distribution, whereas previous HBP methods relied on approximations that could drift into incorrect distributions.
- Efficiency Gains: On synthetic data, the slice sampler converged in about 6.7 minutes, compared to over 52 minutes for the previous approximate Gibbs sampler—a nearly 8x speedup.
- Handling "Negative Transfer": In the NIPS text experiments, simply pooling all data together (without a hierarchy) actually hurt performance once the target dataset got large enough. The R-HBP hierarchy prevented this "negative transfer" by allowing the model to distinguish between shared and source-specific patterns.
- Poisson-Gamma Superiority: For text data, the Poisson-Gamma variant (NHFA-PGM) vastly outperformed Gaussian models and even the popular Hierarchical Dirichlet Process (HDP), suggesting it captures the sparse, "bursty" nature of word counts much better.
Key Results
- Text Modeling: On the NIPS dataset, the proposed NHFA-PGM achieved an average log perplexity of 1102.9, significantly better than HDP (2862.5) and the approximate Gaussian HBP (6232.5).
- Image Retrieval: On the NUS-WIDE dataset, the R-HBP model achieved a Mean Average Precision (MAP) of 0.1945, outperforming several recent state-of-the-art multi-view learning techniques (which hovered around 0.15–0.17).
- Factor Discovery: The model successfully identified the correct number of factors (12) in synthetic tests in under 100 iterations, starting from a single factor.
Limitations and Future Work
- The "Restricted" Constraint: While setting the concentration parameter $\gamma_0 = 1$ enables the slice sampler, it does limit the flexibility of the top-level prior. Future work might explore if this can be relaxed while maintaining tractability.
- Scalability: While faster than previous MCMC methods, the approach still relies on Gibbs sampling, which can be slow for truly massive "Big Data" applications compared to Variational Inference.
- Generalization: The authors note that the slice sampling derivation is general and could potentially be applied to other types of matrix factorizations beyond the ones shown here.
Summary updated