The MediaMill TRECVID 2009 Semantic Video Search Engine

Cees G. M. Snoek, Koen E. A. van de Sande, Ork de Rooij, Bouke Huurnink, Jasper R. R. Uijlings, M. van Liempt, Maarten de Rijke, Jan-Mark Geusebroek, Theo Gevers, Marcel Worring, Arnold W. M. Smeulders, Dennis C. Koelma, Miguel Bugalho, Isabel Trancoso, F. Yan, M. A. Tahir, Krystian Mikolajczyk, Josef KittlerTRECVID 2009 · OpenReview

Abstract

The MediaMill 2009 system addresses the "semantic gap" in video retrieval by scaling concept detection to over one million frames through GPU acceleration and multi-frame analysis. By fusing diverse visual descriptors with audio concepts and employing query-dependent learning to rank, the system achieved top rankings in the TRECVID 2009 benchmark for both concept detection and interactive search.

Introduction

Searching for video content traditionally relies on textual metadata like filenames or transcripts. However, text often fails to capture actual visual content—a challenge known as the "semantic gap." This paper details the MediaMill team's award-winning approach to the TRECVID 2009 benchmark, focusing on robustness in detector accuracy and scalability for massive video archives.

System Architecture

The MediaMill system is built on a modular pipeline that processes both visual and auditory information to bridge the semantic gap.

High-level overview of the MediaMill 2009 system architecture, showing the parallel processing of visual and audio data.

Advanced Concept Detection

The detection pipeline consists of four primary components:

  1. Spatio-Temporal Sampling: Rather than relying on a single keyframe, the system samples up to 10 additional i-frames per shot. It utilizes both salient point detectors (Harris-Laplace) and dense sampling to ensure comprehensive coverage.
  2. Visual Descriptors: A bag-of-words (BoW) approach is employed using a library of color-invariant SIFT descriptors (e.g., OpponentSIFT, C-SIFT) to maintain robustness against lighting and viewpoint changes.
  3. Audio Integration: To complement visual data, 20 audio-specific detectors (e.g., sirens, applause) were integrated into the framework.
  4. GPU Acceleration: To manage the computational load of analyzing 1,000,000 i-frames, the BoW pipeline was implemented on GPUs, resulting in an order-of-magnitude speedup.

Multi-Kernel Learning

The system advances beyond standard Support Vector Machines (SVMs) by exploring sophisticated fusion techniques that learn the optimal way to combine different data views for each specific concept.

The kernel-based learning framework incorporating multiple classifiers like SR-KDA and MK-FDA with episode-constrained cross-validation.

By utilizing Spectral Regression Kernel Discriminant Analysis (SR-KDA) and Multi-Kernel Fisher Discriminant Analysis (MK-FDA), the system determines the most effective weighting for audio versus visual features on a per-concept basis.

Search and Interaction

The search engine supports both automatic and interactive modes:

  • Automatic Search: Employs "learning to rank" (SVM-Rank) to fuse transcript-based, detector-based, and feature-based results.
  • Interactive Search: Utilizes the ForkBrowser, allowing users to navigate video threads based on time, similarity, or concepts. Active Zooming facilitates rapid visual inspection, while Passive Relevance Feedback monitors user behavior to re-rank results in real-time without requiring explicit user input.

Key Insights and Results

The "More is More" principle proved effective; moving to multi-frame processing was a significant performance booster, capturing objects that might be obscured in a single keyframe. Engineering optimizations, such as reusing subregions in descriptors, provided a 16x speedup, making large-scale analysis feasible.

In the TRECVID 2009 benchmark, the MediaMill system achieved:

  • Concept Detection: The top mean inferred Average Precision (infAP of 0.228).
  • Automatic Search: Query-dependent fusion reached a MAP of 0.089.
  • Interactive Search: The MediaMill browsers were the top performers, with the best run achieving an infAP of 0.246.

Conclusion

While the addition of audio concepts provided modest gains, the results highlight the need for more sophisticated audio-visual context modeling. The study concludes that future progress in the field requires moving beyond treating video as a series of still images toward true temporal video analysis focusing on motion and flow.

Summary updated

BibTeX
@inproceedings{snoek2009mediamill,
  author = {Cees G. M. Snoek and Koen E. A. van de Sande and Ork de Rooij and Bouke Huurnink and Jasper R. R. Uijlings and M. van Liempt and Maarten de Rijke and Jan-Mark Geusebroek and Theo Gevers and Marcel Worring and Arnold W. M. Smeulders and Dennis C. Koelma and Miguel Bugalho and Isabel Trancoso and F. Yan and M. A. Tahir and Krystian Mikolajczyk and Josef Kittler},
  title = {{The MediaMill TRECVID 2009 Semantic Video Search Engine}},
  booktitle = {TRECVID},
  year = {2009},
  url = {https://openreview.net/forum?id=bXM-5ygYM6},
}
Figure