Abstract
This paper details the joint participation of the National University of Singapore (NUS) and the Institute for Infocomm Research (I2R) in the TRECVID 2006 challenge. The research addresses the "semantic gap" in video retrieval—the disconnect between low-level visual features and high-level human concepts. By modeling video sequences as "visual sentences" through bi-gram patterns and employing a time-dependent event clustering framework, the team achieved significant improvements in automated search and high-level feature (HLF) extraction. Experimental results on news video archives underscore the efficacy of multimodal fusion and event-centric retrieval strategies.
Introduction
Searching massive video databases presents unique challenges compared to text retrieval due to the high-dimensional, multimodal, and continuous nature of video data. The NUS-I2R team focused on two primary tasks: High-Level Feature (HLF) extraction (detecting concepts like "outdoor" or "person") and Automated Search (finding segments based on user queries). A key hurdle is the temporal structure of news video, where stories often constitute "events" spanning multiple channels and timeframes, a nuance often missed by traditional shot-based retrieval methods.
Methodology
System Architecture and Multimodal Fusion
The proposed system integrates diverse data streams, including Automatic Speech Recognition (ASR), Machine Translation (MT) for multilingual content, Video OCR for on-screen text, and Near-Duplicate Key-frame (NDK) detection. This multimodal approach allows the system to link related stories across different broadcasts and re-rank results based on event context.
Video as Language: The Bi-gram Model
A novel contribution of this work is treating video keyframes as tokens in a "visual sentence." Keyframes are clustered using K-means on color correlograms to create a visual vocabulary. Bi-gram models, typically used in Natural Language Processing, are then applied to identify temporal patterns. This "soft matching" approach enables the identification of sequences even when frames are missing or slightly altered, provided the overall temporal flow remains consistent.
Machine Learning Ensemble and Event Clustering
For HLF extraction, the team utilized an ensemble of Support Vector Machines (SVM), Linear Discriminative Functions (LDF), and Gaussian Mixture Models (GMM). Notably, LDF and GMM were trained using an AUC-maximized algorithm, which proved superior to standard SVMs for certain visual features.
For the search task, the system employs a sliding window approach to group news stories into "events" based on time and location metadata. By clustering these stories using cosine similarity and external knowledge bases like Wikipedia, search results are re-ranked based on their membership in globally significant events.
Experimental Results and Insights
The system was evaluated on the TRECVID 2006 dataset, using the 2005 development set for initial training.
- Search Task Performance: The "Event Model" achieved a Mean Average Precision (MAP) of 0.075, a substantial improvement over the text-only baseline of 0.039. The system performed exceptionally well on "Named Entity" and event-related queries, ranking in the top 10% of international submissions.
- HLF Task Performance: Their best run achieved a MAP of 0.059. The team observed that sequence-level methods (like bi-grams) significantly outperformed frame-level methods for concepts with distinct temporal contexts, such as "Sports" or "Weather."
- Concept Association: The researchers noted that relationships between concepts are often asymmetric (e.g., an "airplane" shot is almost always "outdoor," but the reverse is not true). Quantifying these strengths of association helped prune irrelevant results.
Discussion and Conclusion
The research highlights that while SVMs are powerful, specialized learning algorithms like AUC-maximization can provide a competitive edge for specific visual features. Furthermore, the shift in TRECVID queries toward more visual-oriented "find a shot of X" requests suggests a need for more robust visual modeling.
Limitations identified include the noise inherent in Machine Translation and the fragility of Video OCR. Future work will focus on deeper integration of implicit temporal structures and improving the handling of multilingual, noisy text data to further bridge the semantic gap in large-scale video retrieval.
Summary updated