The Infinite Gamma-Poisson Feature Model

Michalis K. TitsiasNeurIPS 2007 · OpenReview

In the world of unsupervised learning, we often want to describe data as a collection of "latent features." For years, the Indian Buffet Process (IBP) has been the gold standard for this, allowing a model to learn an unbounded number of features. However, the IBP is fundamentally binary: a feature is either present or it isn't.

In this paper, Michalis K. Titsias introduces the Infinite Gamma-Poisson (IGP) Feature Model, a nonparametric Bayesian approach that moves beyond "yes/no" to "how many?" It provides a principled way to model data where features can appear multiple times within a single observation—like multiple cars in a single photograph.

The Problem: Why Counting Matters

Most factorial models assume that a feature occurs at most once per data point. While this works for simple attributes, it fails for structured data like images or text. If you are building a visual object recognition system, knowing that a "car" feature is present is useful, but knowing there are three cars is much better.

Existing solutions often forced a choice: either use a mixture model (which assumes only one cause per data point) or a binary factorial model (which ignores counts). The challenge Titsias tackles is creating a model that is nonparametric (it learns the number of features from the data) and count-based (it tracks how many times each feature appears), all while remaining mathematically tractable for inference.

Methods: From Finite to Infinite

The author derives the model by starting with a finite Gamma-Poisson construction and taking the limit as the number of possible features ($K$) goes to infinity.

  1. The Finite Model: Each data point $n$ has a count $z_{nk}$ for feature $k$. This count is drawn from a Poisson distribution with a feature-specific rate $\lambda_k$. The rates themselves are drawn from a Gamma distribution designed to favor sparsity.
  2. The Infinite Limit: By integrating out the rates ($\lambda_k$) and taking the limit $K \to \infty$, the author defines a distribution over equivalence classes of matrices (specifically, "left-ordered" matrices to handle the infinite columns).
  3. The Stochastic Process: To make the model generative, the paper introduces a sequential process using the Ewens distribution (famous in population genetics). When a new data point arrives, it first decides how many "total occurrences" it has, and then uses the Ewens distribution to partition those occurrences among existing or entirely new features.
  4. Likelihood for Images: The paper pairs this prior with a likelihood model that combines Latent Dirichlet Allocation (LDA) for visual appearance and Gaussian Mixture Models (GMMs) for spatial location. This allows the model to group local image patches into coherent "objects."
Graphical representation of the image likelihood model.

Key Insights and Ideas

  • Sparsity by Design: By using a Gamma prior with a shape parameter $\alpha/K$, the model ensures that as the number of potential features grows, the actual number of active features remains manageable.
  • The Ewens Connection: The paper elegantly links the Gamma-Poisson model to the Ewens distribution. This shows that the IGP is essentially the "count" version of the partitions induced by a Chinese Restaurant Process.
  • Handling "Popularity": The stochastic process naturally incorporates a "rich-get-richer" mechanic. Features that have appeared many times in previous images are more likely to appear in new ones, but there is always a probability of discovering a brand-new feature.

Key Results

The model was evaluated on both synthetic data and the real-world UIUC Cars Database (using SIFT descriptors clustered into a visual vocabulary).

  • Feature Discovery: In synthetic tests containing four distinct regular shapes, the model correctly identified that there were exactly four features in 86% of the MCMC samples. It successfully recovered the shapes and counts of these features despite being initialized with no prior knowledge.
  • Object Localization: On the UIUC car dataset, the model successfully distinguished between "car" features, "tree" features, and "street" features, effectively localizing multiple instances of the same object class within a single image.
Object localization on the UIUC dataset, with ellipses distinguishing cars (red), trees (green), and streets (blue).
  • Semi-Supervised Flexibility: The author demonstrated that you can "nudge" the model by labeling a few images (e.g., drawing bounding boxes around cars), which helps the model anchor its latent features to specific real-world categories.

Limitations and Future Work

While powerful, the model has limitations. Like many nonparametric Bayesian models, inference relies on Markov Chain Monte Carlo (MCMC). Marginalizing out assignments is intractable, requiring importance sampling approximations which can be computationally heavy for very large datasets. Additionally, the current image likelihood model assumes that features (objects) do not occlude one another—an assumption that breaks down in dense, real-world scenes.

Looking forward, the author suggests that this model could be extended to Bayesian matrix factorization, where a matrix of observations is decomposed into two or more non-negative integer matrices. This represents a highly promising direction for collaborative filtering and signal processing.

Summary updated

Abstract

We address the problem of factorial learning which associates a set of latent causes or features with the observed data. Factorial models usually assume that each feature has a single occurrence in a given data point. However, there are data such as images where latent features have multiple occurrences, e.g. a visual object class can have multiple instances shown in the same image. To deal with such cases, we present a probability model over non-negative integer valued matrices with possibly unbounded number of columns. This model can play the role of the prior in an nonparametric Bayesian learning scenario where both the latent features and the number of their occurrences are unknown. We use this prior together with a likelihood model for unsupervised learning from images using a Markov Chain Monte Carlo inference algorithm.
BibTeX
@inproceedings{titsias2007infinite,
  author = {Michalis K. Titsias},
  title = {{The Infinite Gamma-Poisson Feature Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2007},
  url = {https://openreview.net/forum?id=S1EWfKbd-B},
}
Figure