Abstract: This notebook paper describes our system for the untrimmed classification task in the ActivityNet challenge 2016. We investigate multiple state-of-the-art approaches for action recognition in long, untrimmed videos. We exploit hand-crafted motion boundary histogram features as well feature activations from deep networks such as VGG16, GoogLeNet, and C3D. These features are separately fed to linear, one-versus-rest support vector machine classifiers to produce confidence scores for each action class. These predictions are then fused along with the softmax scores of the recent ultra-deep ResNet-101 using weighted averaging.
Introduction
Most action recognition research historically focused on "trimmed" clips—short videos where the action happens from start to finish. However, real-world video (like YouTube) is "untrimmed," meaning a 10-minute video might only contain a few seconds of the actual activity. The authors address the untrimmed classification task, which requires a system to look at a long, messy video and accurately predict which of 200 activity categories (e.g., "polishing shoes" or "playing accordion") are present. The challenge lies in efficiently extracting meaningful signals from long sequences of frames where much of the data might be irrelevant background noise.
The Multi-Stream Framework
The authors developed a multi-stream framework that aggregates predictions from five distinct modules. Instead of building one "super-network," they used a late fusion approach, where each module votes on the activity, and those votes are combined via weighted averaging.
The five streams include:
- MBH (Motion Boundary Histograms): A hand-crafted feature that focuses on local motion. These were encoded using Fisher Vectors to create a global video representation.
- C3D (3D Convolutional Network): Unlike standard 2D nets, this analyzes 16-frame clips to capture spatio-temporal information.
- GoogLeNet: Used to extract frame-level appearance features from the
pool5layer. - VGG16: Frame-level features were extracted and then encoded using VLAD (Vector of Locally Aggregated Descriptors), a technique that helps summarize local features into a single vector.
- ResNet-101: An "ultra-deep" 101-layer network. Interestingly, the authors found that using the raw softmax (probability) scores from this model worked better than extracting intermediate features.
Finally, they applied Multi-class Iterative Re-ranking (MIR) to refine the predictions based on the inherent difficulty of certain video types.
Experimental Setup
The system was evaluated on the ActivityNet Challenge 2016 dataset, which consists of 200 human activity classes across approximately 20,000 real-world YouTube videos (9,942 training, 4,874 validation, and 5,001 test videos).
Results and Key Insights
The team's performance improved steadily as they fused more models together. ResNet-101 was the strongest individual performer (71.81% accuracy), proving that the trend toward deeper architectures in image recognition translates well to the video domain.
- The Power of Complementarity: The hand-crafted MBH features had the lowest individual accuracy (57.32%). However, adding MBH to the deep learning ensemble provided a significant boost. This is because MBH captures low-level motion "texture," while deep networks are better at recognizing high-level objects and scenes.
- Uniform Sampling Wins: In a surprising twist, the authors tried using "action proposals" (algorithms that guess where the action is) to sample frames. They found that simple uniform sampling (picking 25 frames evenly across the video) actually performed better. This suggests that current action proposal methods might be missing relevant frames or that the background context in untrimmed videos is still useful for classification.
Combining all five streams reached 75.14% Top-1 accuracy on the validation set. Their best submission (Run 5) achieved a Mean Average Precision (mAP) of 83.1% and a Top-1 accuracy of 78.44% on the test set.
Conclusion
The authors noted that their attempt to use temporal action proposals to focus the model's attention actually decreased performance. This highlights a gap in how we handle "untrimmed" data; simply ignoring the "non-action" parts of a video might lose valuable contextual clues. Future work likely involves finding more sophisticated ways to integrate temporal information beyond simple average pooling of frames, perhaps through more advanced attention mechanisms or better-integrated spatio-temporal architectures.
Summary updated