A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process

Bahadir Ozdemir, Larry S. DavisNeurIPS 2014

Abstract / TL;DR

This paper introduces the integrative Indian Buffet Process (iIBP), a Bayesian nonparametric framework designed to learn shared latent binary codes for multimodal data, such as images and their corresponding text descriptions. By treating retrieval as a probabilistic inference problem, the model naturally handles cross-modal queries and seamlessly incorporates user relevance feedback, significantly outperforming state-of-the-art hashing techniques.

The Challenge: Bridging the Semantic Gap

In large-scale image retrieval, "hashing" is the gold standard for efficiency. It compresses high-dimensional data into short binary strings so that similar items share similar codes. However, traditional hashing methods often suffer from the semantic gap—the fundamental disconnect between low-level visual features (like pixels and edges) and high-level human meaning (e.g., "a boat on a river").

While multimodal hashing (utilizing both text and images) helps bridge this gap, existing methods face several hurdles:

  1. Rigidity: They typically require a fixed number of bits (features) defined upfront.
  2. Supervision: Many rely heavily on manual class labels, which are not always available and struggle to generalize to unseen categories.
  3. Static Ranking: They often lack a principled mechanism to incorporate real-time user feedback to refine search results.

Methods: The Integrative Latent Feature Model

To address these limitations, the authors propose a generative model built upon the Indian Buffet Process (IBP).

The "Buffet" Metaphor

In the IBP, images act as "customers" and latent features are "dishes" in an infinitely long buffet. Each customer selects a set of dishes (binary features). The primary advantage of this nonparametric approach is that the number of features ($K$) is not fixed; instead, it is learned directly from the data's complexity.

Key Technical Components

  • Linear-Gaussian Model: The model posits that an image's visual and textual components are generated by multiplying its latent binary code ($Z$) with a feature-value matrix ($A$), plus some Gaussian noise.
Matrix factorization view of the Integrative Latent Feature Model, showing how a shared latent binary code generates both visual and textual features.
  • Probabilistic Retrieval: Rather than merely calculating the Hamming distance between binary codes, the authors extend the probabilistic metaphor. A query is treated as a new "customer," and the system calculates the probability that existing images are "friends" of the query based on shared latent features.
  • Relevance Feedback as "Pseudo-Images": When a user marks an image as relevant, the model treats this feedback as imaginary data points (pseudo-images). This biases the latent space distribution, effectively nudging the search results toward the user's specific intent.

Datasets

The model was rigorously evaluated on two popular multimodal benchmarks:

  1. PASCAL-Sentence 2008: Contains 1,000 images across 20 categories, each paired with five descriptive sentences.
  2. SUN-Attribute: A larger, more challenging dataset containing 14,340 images from 717 categories, annotated with 102 different attributes (e.g., "materials," "spatial envelope").

Key Insights and Ideas

  • Joint Latent Space: By forcing the model to find a single binary code that explains both the text and the pixels, the latent features naturally become more "semantic." For instance, if an image has a "car" feature, it must explain both the visual shape of the car and the presence of the word "automobile."
  • Flexibility via Nonparametrics: Because it utilizes the IBP, the system adapts its complexity to the dataset. It avoids forcing a 64-bit representation if the data only justifies 23 meaningful features.
  • Expectation over Sampling: The authors discovered that using the conditional expectation of relevance to rank images is more stable and accurate than simply sampling binary values.
  • Unified Framework: The same underlying mathematics handles image-to-image search, text-to-image search, and user feedback, resulting in an architecturally elegant system.

Key Results

The iIBP method was benchmarked against several leading models, including Multimodal Deep Boltzmann Machines (mDBM) and Predictable Dual-View Hashing (PDH).

Precision-Recall curves comparing iIBP against state-of-the-art hashing methods on PASCAL-Sentence and SUN datasets.
  • Superior Precision: On the PASCAL-Sentence dataset, iIBP consistently achieved higher mean precision across all recall levels compared to state-of-the-art methods.
  • Handling Sparsity: On the SUN-Attribute dataset—where there are very few positive matches per category—iIBP significantly outperformed unimodal hashing methods, which typically struggle when visual similarity does not align with semantic categories.
  • Feedback Boost: Incorporating relevance feedback provided a clear improvement in the precision-recall curves, demonstrating that the model successfully learns from user interaction.
  • Efficiency: Despite the inherent complexity of Bayesian inference, a typical query took less than 1 second using their optimized Matlab implementation.

Limitations and Future Work

While highly effective, the model relies on Markov chain Monte Carlo (MCMC) sampling for inference. Although the authors utilized an accelerated version, MCMC can still be computationally demanding as datasets scale to millions of entries.

The authors suggest two primary paths forward:

  1. Variational Inference: Transitioning from MCMC to deterministic variational methods (such as truncated stick-breaking approximations) could further accelerate the learning process.
  2. User Inference: Developing a more sophisticated model to automatically infer user intent from subtler, implicit feedback cues.

Summary updated

BibTeX
@inproceedings{ozdemir2014probabilistic,
  author = {Bahadir Ozdemir and Larry S. Davis},
  title = {{A Probabilistic Framework for Multimodal Retrieval using Integrative Indian Buffet Process}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure