Abstract
High-level event classification in videos often suffers from a "semantic gap" between low-level visual features and complex human activities. We present ACTIVE (Activity Concept Transitions in Video Events), a framework that models the temporal flow of mid-level activity concepts. By treating a video as a sequence of clips corresponding to latent states in a Hidden Markov Model (HMM) and applying Fisher Kernel techniques, we derive a compact HMM Fisher Vector (HMMFV). This descriptor efficiently encodes temporal dynamics and performs robustly even with limited training data, outperforming traditional pooling methods on the NIST TRECVID Multimedia Event Detection dataset.
The Semantic Gap in Video Classification
Classifying complex "in-the-wild" events—such as a "wedding ceremony" or "changing a tire"—is a significant challenge in computer vision. Traditional "bag-of-words" models aggregate low-level features (e.g., SIFT or motion descriptors) into a single histogram. This approach has two critical weaknesses:
- Loss of Temporal Order: It ignores the sequence of actions. In a wedding, a kiss typically precedes a hug; a bag-of-words model treats these as unordered occurrences.
- Semantic Gap: There is a massive disconnect between raw pixel gradients and high-level concepts like "birthday party."
The ACTIVE Framework
The ACTIVE pipeline bridges this gap by explicitly modeling transitions between mid-level "activity concepts" like walking, hugging, or dancing.
The process involves four key steps:
- Clip-level Concept Detection: A video is decomposed into short clips. Pre-trained classifiers estimate the presence of various atomic activity concepts for each clip.
- HMM Modeling: A Hidden Markov Model is employed where latent states represent these activity concepts and observations are the clip-level features.
- Fisher Vector Encoding (HMMFV): Instead of standard HMM inference, the authors compute the derivative of the log-likelihood with respect to the HMM's transition parameters. This generates a fixed-length vector—the HMMFV—that describes how a specific video's transitions deviate from a global model.
- Classification: These HMMFV descriptors are used as input for a discriminative classifier, such as a Support Vector Machine (SVM).
Key Insights: Transitions as Features
The core innovation of ACTIVE is the realization that the change from one state to another is often more discriminative than the states themselves.
By using Fisher Kernels, the framework captures these transitions efficiently. High values in the HMMFV indicate transitions that are significantly more likely in a specific video than in a generic model, providing a robust signature for complex events. Furthermore, reasoning about mid-level concepts rather than raw pixels makes the system more resilient to the noise typical of amateur web videos.
Experimental Results
The researchers evaluated ACTIVE on the NIST TRECVID MED 11 dataset and used UCF 101 for cross-domain concept training. Key findings include:
- Superior Temporal Modeling: HMMFV outperformed standard Max Pooling by 11.9% in mean Average Precision (mAP).
- State-of-the-Art Accuracy: ACTIVE surpassed existing joint modeling approaches in 11 out of 15 event categories, reaching a mAP of 70.8% when fused with low-level features.
- Robustness to Small Data: In scenarios with only 10 positive training samples, HMMFV maintained a mAP of 0.562, significantly higher than the 0.421 achieved by low-level feature methods.
- Generalization: The framework performed well even when using concept detectors trained on a different dataset (UCF 101), demonstrating its flexibility.
Conclusion
ACTIVE demonstrates that modeling the temporal transitions of mid-level activity concepts is a powerful strategy for high-level video event classification. By combining HMMs with Fisher Kernel encoding, the framework provides a scalable and robust solution that effectively bridges the semantic gap while preserving the vital temporal structure of human activities.
Summary updated