Content-based Video Indexing and Retrieval Using Corr-LDA

Rahul Radhakrishnan Iyer, Sanjeel Parekh, Vikas Mohandoss, Anush Ramsurat, Bhiksha Raj, Rita SingharXiv 2016 · arXiv

Abstract

This paper presents a novel application of the Correspondence-Latent Dirichlet Allocation (Corr-LDA) model for content-based video indexing and retrieval. By representing audio tracks as "bags of auditory words" and employing a multinomial distribution within a generative probabilistic framework, the proposed method effectively bridges the semantic gap between low-level multimedia features and high-level textual queries. Experimental results on the TRECVID MED11 dataset demonstrate that an audio-only Corr-LDA approach can outperform traditional visual-feature-based SVM classifiers, highlighting the significant semantic value of auditory cues in video search tasks.

Introduction: The Semantic Gap in Video Retrieval

Traditional video search engines rely heavily on user-provided metadata and tags. However, this "sparse tagging" approach is often unreliable due to inconsistent or missing labels. To overcome these limitations, researchers have turned toward Content-Based Retrieval (CBR), which analyzes the actual pixels and audio within a video.

The primary challenge in CBR is the "semantic gap"—the difficulty of mapping low-level machine representations (such as audio frequencies or color histograms) to high-level human concepts (such as "grooming an animal"). This paper addresses this gap by developing a probabilistic model designed to automatically learn the hidden relationships between multimedia content and textual descriptors.

Conceptual framework for modeling the correspondence between multimedia content and text descriptors.

Methodology: Adapting Corr-LDA for Audio

The authors adapt the Correspondence-Latent Dirichlet Allocation (Corr-LDA) model, a generative framework originally designed for image-captioning tasks. Their approach introduces several key modifications to optimize the model for video retrieval:

  1. Audio-Centric Representation: To evaluate the descriptive power of sound and reduce computational overhead, the model focuses exclusively on audio tracks. Audio is processed into "Acoustic Unit Descriptors" (AUDs).
  2. Bag-of-Words (BoW) Framework: Both audio features and text queries are treated as unordered "bags of words," allowing for a unified representation of disparate data types.
  3. Multinomial Distribution: While the original Corr-LDA used a Gaussian distribution for visual features, the authors utilize a multinomial distribution to better fit the discrete frequency data of the BoW audio representation.
  4. Latent Topic Modeling: The model assumes that a video's audio and its text annotations are generated from a shared set of latent "topics." By learning these topics through Variational Expectation Maximization (EM), the system can calculate the probability that a specific text query matches an unannotated video.
Plate notation representing the probabilistic graphical models used for latent topic discovery and correspondence modeling.

Experimental Results

The model was benchmarked using a subset of the TRECVID MED11 dataset, containing 500 training and 125 test videos across five categories: repairing an appliance, parade, parkour, grooming an animal, and making a sandwich.

Retrieval Performance

The Corr-LDA model, when configured with 100 latent topics, achieved a Mean Average Precision (MAP@10) of 0.243. This significantly outperformed visual-only Support Vector Machine (SVM) baselines, such as SVM-CTS (0.160) and SVM-DSIFT (0.121). This result suggests that audio alone can provide more robust semantic cues for complex event detection than certain standard visual features.

Indexing and Annotation Quality

The quality of auto-generated tags was evaluated using perplexity and F-scores. As the number of topics increased from 5 to 100, the model's perplexity improved (dropped) from 43.37 to 40.59. Simultaneously, the F-score for generated annotations nearly doubled, rising from 0.030 to 0.051, indicating that higher topic granularity allows for more precise concept matching.

Conclusion and Future Directions

This research demonstrates that audio is a "semantic heavy hitter" in the realm of video retrieval. By leveraging shared latent spaces, Corr-LDA effectively bypasses the need for hard-coded mappings between signals and semantics.

However, limitations remain, particularly regarding audio ambiguity (e.g., distinguishing a parade from a sporting event based on crowd noise) and the model's inability to handle out-of-vocabulary (OOV) words. Future work will focus on integrating visual BoW features to create a multimodal Corr-LDA model and implementing smoothing techniques to improve the handling of unseen textual queries.

Summary updated

Abstract

Existing video indexing and retrieval methods on popular web-based multimedia sharing websites are based on user-provided sparse tagging. This paper proposes a very specific way of searching for video clips, based on the content of the video. We present our work on Content-based Video Indexing and Retrieval using the Correspondence-Latent Dirichlet Allocation (corr-LDA) probabilistic framework. This is a model that provides for auto-annotation of videos in a database with textual descriptors, and brings the added benefit of utilizing the semantic relations between the content of the video and text. We use the concept-level matching provided by corr-LDA to build correspondences between text and multimedia, with the objective of retrieving content with increased accuracy. In our experiments, we employ only the audio components of the individual recordings and compare our results with an SVM-based approach.
BibTeX
@misc{iyer2016content,
  author = {Rahul Radhakrishnan Iyer and Sanjeel Parekh and Vikas Mohandoss and Anush Ramsurat and Bhiksha Raj and Rita Singh},
  title = {{Content-based Video Indexing and Retrieval Using Corr-LDA}},
  year = {2016},
  eprint = {1602.08581},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.08581},
}
Figure