Abstract
In the TRECVID 2014 evaluation, the Carnegie Mellon University (CMU) Informedia team demonstrated state-of-the-art performance in large-scale video analysis. Their approach integrated a novel self-paced learning framework for concept detection, deep convolutional neural network (DCNN) features, and significant pipeline optimizations. This report details the methodologies that led to top rankings in Multimedia Event Detection (MED), Semantic Indexing (SIN), and Surveillance Event Detection (SED), including a 38x speedup in search efficiency and a 51.2% boost in zero-example search performance through self-paced reranking.
Introduction
The primary challenge addressed by the Informedia team is the automatic analysis, indexing, and search of massive video databases. The TRECVID 2014 evaluation set forth several rigorous tasks: Multimedia Event Detection (MED) and Recounting (MER), which involve finding specific events with minimal or no video examples; Semantic Indexing (SIN), requiring the automatic tagging of video shots; and Surveillance Event Detection (SED), focusing on activity spotting in CCTV footage. Beyond accuracy, the team addressed the critical sub-problem of computational efficiency, aiming to make complex machine learning models viable for real-world, interactive applications.
Methodology
The CMU system utilized a multi-modal pipeline characterized by several innovative techniques:
- Self-Paced Learning (SPL): To train over 3,000 semantic concept detectors, the team employed SPL, which organizes training samples from "easy" to "complex," mimicking human learning patterns.
- Self-Paced Reranking (SPaR): For zero-example searches, SPaR acts as a pseudo-relevance feedback mechanism. It assumes top-retrieved videos are "easy" positives and iteratively updates the model to identify more difficult relevant samples.
- Deep Features and Trajectories: The team used 15 different DCNN features pre-trained on ImageNet and enhanced standard motion-tracking features (Improved Dense Trajectories) by encoding them into Fisher Vectors.
- Extreme Optimization: To achieve interactive speeds, the team replaced slow kernel SVMs with linear classifiers via Explicit Feature Map (EFM) approximations and utilized 32X Product Quantization (PQ) to reduce I/O bottlenecks.
Datasets and Features
The system was evaluated on the MED14-Eval Full set (200,000 videos), MEDTEST14, and IACC datasets. External data played a crucial role: ImageNet was used for deep learning features, while YFCC100M, UCF101, and various instructional videos provided the basis for training the extensive vocabulary of 3,000+ semantic concept detectors. Additionally, the IEMOCAP database was used to train acoustic-based emotion classifiers.
Key Results and Insights
The Informedia team achieved leading performance across all MED settings. A standout result was the effectiveness of SPaR, which boosted 0-Example Mean Average Precision (MAP) by 51.2% for ad-hoc events. Furthermore, the team demonstrated that the gap between zero-example (text-only) and 10-example search is shrinking, largely due to the massive vocabulary of semantic concepts.
Efficiency gains were equally impressive. By approximating complex models and compressing data, the team achieved a 38x speedup in Event Search prediction time. Searching 200,000 videos with 47 features was reduced from 90 minutes to under 2.5 minutes on a single workstation, with only a negligible 3% drop in relative accuracy.
Limitations and Future Work
Despite these successes, the authors identified vulnerabilities in reranking. If an initial text query retrieves off-topic videos, SPaR can confidently learn incorrect visual concepts. Additionally, translating human text queries into system-readable semantic weights (Semantic Query Generation) remains a largely manual, ad-hoc process. Future research will focus on automating this query translation and reducing the computational costs of advanced surveillance detection methods, such as the "template bounding box" approach.
Summary updated