TL;DR
This paper introduces ERMH-BoW, a novel feature for video event detection that combines "what" (visual content) and "how" (motion) aspects of an event. By leveraging motion relativity and visual relatedness, ERMH-BoW effectively addresses challenges like camera motion distortion and visual word correlation, leading to significant performance improvements over existing methods.
What Problem Does This Paper Address?
Detecting and recognizing events in video content is a crucial but challenging task. Existing methods often struggle because they either focus solely on the visual content ("what") or the motion ("how") of an event, failing to capture a complete description. Furthermore, motion analysis in unconstrained videos is hampered by noisy motion distributions and distortions caused by camera movement, making it difficult to accurately represent object activities. This paper aims to bridge this gap by proposing a feature that integrates visual and motion information more effectively, while also being robust to camera motion and visual word ambiguities.
Methods
The core of this paper is the proposed Expanded Relative Motion Histogram of Bag-of-Visual-Words (ERMH-BoW) feature. Here's a breakdown of its key components:
- Bag-of-Visual-Words (BoW): This standard technique is used to represent the "what" aspect of an event. Local image features (SIFT descriptors) are clustered to form a visual vocabulary, and each frame is represented as a histogram of these visual words.
- Motion Histogram of BoW (MH-BoW): To capture the "how" aspect, local motion information of keypoints mapped to each visual word is aggregated into a 4-directional histogram. This integrates "what" and "how" but is still susceptible to camera motion.
- Relative Motion Histogram of BoW (RMH-BoW): To address camera motion distortion, the paper proposes using relative motion between visual words. Instead of absolute motion, the motion of one visual word is measured with respect to another. This provides a more honest representation of object activities, as demonstrated in Figure 2.
- Expansion with Visual Relatedness (ERMH-BoW): A key innovation is the expansion of RMH-BoW using visual relatedness. Visual words that are semantically similar (e.g., representing the same object category) but might be distinct in the BoW vocabulary can lead to feature mismatch. The paper uses a visual ontology and the JCN metric to calculate visual relatedness and then diffuses the relative motion histograms between correlated visual words. This alleviates the visual word correlation problem.
For event detection, ERMH-BoW features are used to compute pairwise distances between video clips using Earth Mover's Distance (EMD). These distances are then fed into a Support Vector Machine (SVM) with a Gaussian kernel for classification.
Datasets
The experiments were conducted on the LSCOM annotated events dataset, specifically using data from the TRECVID 2005 benchmark. The dataset was re-annotated for events, and 14 events with sufficient positive examples were selected for the experiments. The dataset comprises approximately 40,000 clips, with 50% used for training and 50% for testing.
Key Insights and Ideas
- Motion Relativity is Key for Robustness: The paper convincingly argues that relative motion, rather than absolute motion or motion distribution, is crucial for accurately describing object activities in the presence of varying camera movements. This is a significant insight for unconstrained video analysis.
- Integrating "What" and "How" is Essential: The success of ERMH-BoW highlights the importance of combining visual content (what objects/scenes are present) with their dynamic interactions (how they move). This holistic approach provides a more complete event description.
- Visual Relatedness as a "Glue": The expansion step using visual relatedness is a clever way to overcome the limitations of discrete visual words in BoW. It allows for a more nuanced representation by acknowledging that different visual words might refer to similar concepts, thus improving matching robustness.
- Local Motion is More Discriminative: Moving away from global motion distribution towards local motion of visual words makes the features more discriminative, focusing on the motion of relevant objects rather than noisy background motion.
Key Results
- Significant Performance Improvement: ERMH-BoW achieved a Mean Average Precision (MAP) of 25.96% on the LSCOM dataset, which is a substantial improvement over:
- Grid-based Motion Histogram (GMH): 14.90% (74.2% improvement)
- Bag-of-Visual-Words (BoW): 20.86% (24.4% improvement)
- Motion Histogram of BoW (MH-BoW): 22.66% (14.6% improvement over BoW, 9.0% improvement over MH-BoW)
- Relative Motion Histogram of BoW (RMH-BoW): 24.69% (5.1% improvement over RMH-BoW)
- Sequence vs. Keyframe: Sequence-based approaches (like ERMH-BoW) significantly outperformed keyframe-based methods, demonstrating the value of temporal information for event detection.
- Motion-Intensive Events Benefit Most: Improvements were most pronounced for motion-intensive events like Running, Walking, and People_Marching.
Limitations and Future Work
- Computational Cost: The paper acknowledges that sequence-based feature extraction and learning are naturally slower than keyframe-based methods. Their current implementation is approximately 10 times slower.
- Non-Motion-Intensive Events: For events with little or no motion (e.g., Singing, People_Crying), motion-based features showed less improvement or even a slight decline compared to static features. This suggests that for such events, static or semantic features might be more suitable.
- Future Work: The authors suggest researching more efficient models for utilizing motion relativity and visual relatedness for video event detection, aiming to reduce the computational cost. They also imply a need for adaptive feature selection based on event type.
Summary updated