Abstract
The MediaMill 2009 system addresses the "semantic gap" in video retrieval by scaling concept detection to over one million frames through GPU acceleration and multi-frame analysis. By fusing diverse visual descriptors with audio concepts and employing query-dependent learning to rank, the system achieved top rankings in the TRECVID 2009 benchmark for both concept detection and interactive search.
Introduction
Searching for video content traditionally relies on textual metadata like filenames or transcripts. However, text often fails to capture actual visual content—a challenge known as the "semantic gap." This paper details the MediaMill team's award-winning approach to the TRECVID 2009 benchmark, focusing on robustness in detector accuracy and scalability for massive video archives.
System Architecture
The MediaMill system is built on a modular pipeline that processes both visual and auditory information to bridge the semantic gap.
Advanced Concept Detection
The detection pipeline consists of four primary components:
- Spatio-Temporal Sampling: Rather than relying on a single keyframe, the system samples up to 10 additional i-frames per shot. It utilizes both salient point detectors (Harris-Laplace) and dense sampling to ensure comprehensive coverage.
- Visual Descriptors: A bag-of-words (BoW) approach is employed using a library of color-invariant SIFT descriptors (e.g., OpponentSIFT, C-SIFT) to maintain robustness against lighting and viewpoint changes.
- Audio Integration: To complement visual data, 20 audio-specific detectors (e.g., sirens, applause) were integrated into the framework.
- GPU Acceleration: To manage the computational load of analyzing 1,000,000 i-frames, the BoW pipeline was implemented on GPUs, resulting in an order-of-magnitude speedup.
Multi-Kernel Learning
The system advances beyond standard Support Vector Machines (SVMs) by exploring sophisticated fusion techniques that learn the optimal way to combine different data views for each specific concept.
By utilizing Spectral Regression Kernel Discriminant Analysis (SR-KDA) and Multi-Kernel Fisher Discriminant Analysis (MK-FDA), the system determines the most effective weighting for audio versus visual features on a per-concept basis.
Search and Interaction
The search engine supports both automatic and interactive modes:
- Automatic Search: Employs "learning to rank" (SVM-Rank) to fuse transcript-based, detector-based, and feature-based results.
- Interactive Search: Utilizes the ForkBrowser, allowing users to navigate video threads based on time, similarity, or concepts. Active Zooming facilitates rapid visual inspection, while Passive Relevance Feedback monitors user behavior to re-rank results in real-time without requiring explicit user input.
Key Insights and Results
The "More is More" principle proved effective; moving to multi-frame processing was a significant performance booster, capturing objects that might be obscured in a single keyframe. Engineering optimizations, such as reusing subregions in descriptors, provided a 16x speedup, making large-scale analysis feasible.
In the TRECVID 2009 benchmark, the MediaMill system achieved:
- Concept Detection: The top mean inferred Average Precision (infAP of 0.228).
- Automatic Search: Query-dependent fusion reached a MAP of 0.089.
- Interactive Search: The MediaMill browsers were the top performers, with the best run achieving an infAP of 0.246.
Conclusion
While the addition of audio concepts provided modest gains, the results highlight the need for more sophisticated audio-visual context modeling. The study concludes that future progress in the field requires moving beyond treating video as a series of still images toward true temporal video analysis focusing on motion and flow.
Summary updated