In the world of unsupervised learning, we often want to describe data as a collection of "latent features." For years, the Indian Buffet Process (IBP) has been the gold standard for this, allowing a model to learn an unbounded number of features. However, the IBP is fundamentally binary: a feature is either present or it isn't.
In this paper, Michalis K. Titsias introduces the Infinite Gamma-Poisson (IGP) Feature Model, a nonparametric Bayesian approach that moves beyond "yes/no" to "how many?" It provides a principled way to model data where features can appear multiple times within a single observation—like multiple cars in a single photograph.
The Problem: Why Counting Matters
Most factorial models assume that a feature occurs at most once per data point. While this works for simple attributes, it fails for structured data like images or text. If you are building a visual object recognition system, knowing that a "car" feature is present is useful, but knowing there are three cars is much better.
Existing solutions often forced a choice: either use a mixture model (which assumes only one cause per data point) or a binary factorial model (which ignores counts). The challenge Titsias tackles is creating a model that is nonparametric (it learns the number of features from the data) and count-based (it tracks how many times each feature appears), all while remaining mathematically tractable for inference.
Methods: From Finite to Infinite
The author derives the model by starting with a finite Gamma-Poisson construction and taking the limit as the number of possible features ($K$) goes to infinity.
- The Finite Model: Each data point $n$ has a count $z_{nk}$ for feature $k$. This count is drawn from a Poisson distribution with a feature-specific rate $\lambda_k$. The rates themselves are drawn from a Gamma distribution designed to favor sparsity.
- The Infinite Limit: By integrating out the rates ($\lambda_k$) and taking the limit $K \to \infty$, the author defines a distribution over equivalence classes of matrices (specifically, "left-ordered" matrices to handle the infinite columns).
- The Stochastic Process: To make the model generative, the paper introduces a sequential process using the Ewens distribution (famous in population genetics). When a new data point arrives, it first decides how many "total occurrences" it has, and then uses the Ewens distribution to partition those occurrences among existing or entirely new features.
- Likelihood for Images: The paper pairs this prior with a likelihood model that combines Latent Dirichlet Allocation (LDA) for visual appearance and Gaussian Mixture Models (GMMs) for spatial location. This allows the model to group local image patches into coherent "objects."
Key Insights and Ideas
- Sparsity by Design: By using a Gamma prior with a shape parameter $\alpha/K$, the model ensures that as the number of potential features grows, the actual number of active features remains manageable.
- The Ewens Connection: The paper elegantly links the Gamma-Poisson model to the Ewens distribution. This shows that the IGP is essentially the "count" version of the partitions induced by a Chinese Restaurant Process.
- Handling "Popularity": The stochastic process naturally incorporates a "rich-get-richer" mechanic. Features that have appeared many times in previous images are more likely to appear in new ones, but there is always a probability of discovering a brand-new feature.
Key Results
The model was evaluated on both synthetic data and the real-world UIUC Cars Database (using SIFT descriptors clustered into a visual vocabulary).
- Feature Discovery: In synthetic tests containing four distinct regular shapes, the model correctly identified that there were exactly four features in 86% of the MCMC samples. It successfully recovered the shapes and counts of these features despite being initialized with no prior knowledge.
- Object Localization: On the UIUC car dataset, the model successfully distinguished between "car" features, "tree" features, and "street" features, effectively localizing multiple instances of the same object class within a single image.
- Semi-Supervised Flexibility: The author demonstrated that you can "nudge" the model by labeling a few images (e.g., drawing bounding boxes around cars), which helps the model anchor its latent features to specific real-world categories.
Limitations and Future Work
While powerful, the model has limitations. Like many nonparametric Bayesian models, inference relies on Markov Chain Monte Carlo (MCMC). Marginalizing out assignments is intractable, requiring importance sampling approximations which can be computationally heavy for very large datasets. Additionally, the current image likelihood model assumes that features (objects) do not occlude one another—an assumption that breaks down in dense, real-world scenes.
Looking forward, the author suggests that this model could be extended to Bayesian matrix factorization, where a matrix of observations is decomposed into two or more non-negative integer matrices. This represents a highly promising direction for collaborative filtering and signal processing.
Summary updated