A Slice Sampler for Restricted Hierarchical Beta Process with Applications to Shared Subspace Learning

Sunil Kumar Gupta, Dinh Q. Phung, Svetha Venkatesh

Abstract

Hierarchical beta process has found interesting applications in recent years. In this paper we present a modified hierarchical beta process prior with applications to hierarchical modeling of multiple data sources. The novel use of the prior over a hierarchical factor model allows factors to be shared across different sources. We derive a slice sampler for this model, enabling tractable inference even when the likelihood and the prior over parameters are non-conjugate. This allows the application of the model in much wider contexts without restrictions. We present two different data generative models a linear Gaussian-Gaussian model for real valued data and a linear Poisson-gamma model for count data. Encouraging transfer learning results are shown for two real world applications text modeling and content based image retrieval.

Introduction

In Bayesian multi-source modeling, a primary objective is to discover latent patterns, or "factors," that are shared across disparate datasets. For example, in text analysis, different research domains might share a common set of topics. The Hierarchical Beta Process (HBP) is a powerful framework for this, as it allows for an infinite library of factors from which each data source can select a subset.

However, the HBP faces a significant inference bottleneck. Standard approaches typically require conjugacy—a mathematical convenience where prior and likelihood distributions match—to maintain tractability. Without it, researchers often resort to arbitrary truncation (manually capping factor counts) or crude approximations. This paper introduces the Restricted Hierarchical Beta Process (R-HBP) and a novel slice sampler that enables exact, tractable inference for hierarchical factor analysis even in non-conjugate settings.

Methodology

The authors propose two main innovations to overcome the inference bottleneck: the R-HBP prior and a corresponding slice sampler.

Restricted Hierarchical Beta Process (R-HBP)

By fixing the concentration parameter of the parent beta process to unity ($\gamma_0 = 1$), the authors link the HBP to the Indian Buffet Process (IBP) and its stick-breaking construction. This restriction allows for the sharing of an infinite number of latent factors across multiple data sources in a structured manner.

Graphical model of the Restricted Hierarchical Beta Process (R-HBP) showing the hierarchy and the auxiliary slice variable $\rho$.

The Slice Sampler

The "secret sauce" of the methodology is the slice sampler. Instead of managing an infinite number of factors, the sampler introduces an auxiliary "slice" variable ($\rho$). In each iteration, this variable effectively "slices" the infinite process, leaving only a finite number of active factors to be computed. This makes the infinite representation tractable without needing to pre-specify a truncation limit.

This framework is applied to two specific models:

  • NHFA-PGM: A Poisson-Gamma model designed for count data (e.g., word frequencies).
  • NHFA-GGM: A Gaussian-Gaussian model for real-valued data (e.g., image features).

Experiments and Results

The model was evaluated on synthetic data and two real-world datasets: the NIPS 0-12 text collection and the NUS-WIDE animal image dataset.

Synthetic Data and Efficiency

On synthetic data with 100-dimensional binary factors, the slice sampler successfully identified the correct number of factors (12) in under 100 iterations, starting from a single factor. Notably, the slice sampler achieved a nearly 8x speedup over previous approximate Gibbs samplers, converging in 6.7 minutes compared to 52 minutes.

Synthetic data results: (a) True factors, (b) Inferred factors, and (c) Convergence of the number of active factors to the true value (12).

Real-World Performance

  • Text Modeling: On the NIPS dataset, the NHFA-PGM achieved an average log perplexity of 1102.9, significantly outperforming the Hierarchical Dirichlet Process (2862.5) and approximate Gaussian HBP (6232.5). This suggests the Poisson-Gamma variant better captures the sparse, "bursty" nature of word counts.
  • Image Retrieval: On the NUS-WIDE dataset, the R-HBP model achieved a Mean Average Precision (MAP) of 0.1945, outperforming several state-of-the-art multi-view learning techniques which typically ranged between 0.15 and 0.17.

Discussion and Conclusion

The R-HBP slice sampler provides "exact" inference by sampling from the true posterior distribution, avoiding the drift common in previous approximations. The hierarchical structure also prevents "negative transfer" by allowing the model to distinguish between shared and source-specific patterns.

While the $\gamma_0 = 1$ constraint enables tractability, it does limit the flexibility of the top-level prior. Furthermore, while faster than previous MCMC methods, the reliance on Gibbs sampling may still face scalability issues for massive datasets compared to variational inference. Future work may explore relaxing these constraints or applying the slice sampling derivation to broader matrix factorization contexts.

Summary updated

Abstract

Hierarchical beta process has found interesting applications in recent years. In this paper we present a modified hierarchical beta process prior with applications to hierarchical modeling of multiple data sources. The novel use of the prior over a hierarchical factor model allows factors to be shared across different sources. We derive a slice sampler for this model, enabling tractable inference even when the likelihood and the prior over parameters are non-conjugate. This allows the application of the model in much wider contexts without restrictions. We present two different data generative models a linear GaussianGaussian model for real valued data and a linear Poisson-gamma model for count data. Encouraging transfer learning results are shown for two real world applications text modeling and content based image retrieval.
BibTeX
@inproceedings{gupta2012slice,
  author = {Sunil Kumar Gupta and Dinh Q. Phung and Svetha Venkatesh},
  title = {{A Slice Sampler for Restricted Hierarchical Beta Process with Applications to Shared Subspace Learning}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2012},
  url = {https://openreview.net/forum?id=TRUNafIAH8v},
}
Figure