Abstract
The BUPT-MCPRL team secured top-three finishes in the TRECVID 2018 competition by deploying a sophisticated multi-stage pipeline for video activity detection and instance search. Their approach combines standard object detection (Faster R-CNN, YOLO) with specialized temporal models like Temporal Relation Networks (TRN) and 3D-ResNets. By fusing trajectory-based heuristics with deep temporal classification, the system effectively manages the computational complexity of surveillance-style video analysis.
Introduction
Large-scale video analysis in surveillance contexts presents two primary challenges: Activities in Extended Video (ActEV) and Instance Search (INS). ActEV requires detecting and localizing human-vehicle interactions (e.g., "opening a trunk") and person-only activities (e.g., "talking on a phone") within long, unedited footage. This is a "needle in a haystack" problem with high computational costs. INS involves finding a specific person at a specific location across a massive video library, necessitating a simultaneous understanding of identity (who) and scene (where).
Methodology
ActEV Pipeline
The BUPT-MCPRL team implemented a "detect-track-propose-classify" architecture to handle the ActEV task.
- Detection & Tracking: Pedestrians are detected using Faster R-CNN with a ResNet-50 backbone and Online Hard Example Mining (OHEM), while vehicles are identified via YOLO. Tracking is enhanced through deep Re-Identification (Re-ID) features, which provide greater robustness against occlusions and lighting changes compared to traditional color histograms.
- Event Proposals: To reduce computational overhead, the system generates proposals using hand-crafted rules. For instance, human-vehicle interactions are only proposed when the Intersection over Union (IoU) between a person and a vehicle exceeds a specific threshold.
- Specialized Classification: Events are classified using models tailored to their nature:
- Trajectory Analysis: Used for vehicle turning events by analyzing angle variations.
- Temporal Models: Convolutional-De-Convolutional (CDC) networks provide precise temporal localization for "Opening/Closing" events, while Temporal Relation Networks (TRN) capture the logical sequence of actions like "Open_Trunk" vs. "Closing_Trunk."
- 3D-ResNets: Employed for complex actions such as "Entering/Exiting" and "Talking."
Instance Search (INS) Pipeline
The INS system utilizes a dual-path retrieval strategy:
- Location Path: Combines local features (RootSIFT, Bag-of-Words) with global features (ResNeXt trained on Places365).
- Person Path: Uses MTCNN for face detection and cosine distance for matching, supplemented by searching video transcripts for character names.
- Fusion: The system performs "Rank Fusion," combining results from (Location → Person) and (Person → Location) searches. A Random Forest classifier, trained on TRECVID 2016 data, provides a final re-ranking boost.
Key Insights
A central theme of the BUPT-MCPRL approach is the use of heuristics as a filter. By applying simple physical rules (velocity thresholds, proximity) to generate proposals, the team significantly reduced the search space. This allowed for the targeted application of computationally expensive 3D-CNNs. Furthermore, the team distinguished between "states" (discriminated by pose/context, like "Riding") and "actions" (discriminated by motion patterns, like "Closing"), selecting the optimal tool (image vs. temporal classification) for each.
Results and Discussion
The system performed exceptionally well, earning 3rd place on the ActEV Activity Detection (AD) leaderboard and 2nd place on the Activity and Object Detection (AOD) leaderboard.
In the INS task, the team achieved a maximum Mean Average Precision (mAP) of 44.7 for interactive search runs and 38.0 mAP for automatic runs. Despite these successes, the authors noted difficulties with "small movement" events like "Talking" or "Interacts," which are often hard to distinguish from background noise.
Conclusion
The BUPT-MCPRL team demonstrated that a multi-stage pipeline combining robust detection, heuristic-based proposals, and specialized temporal classification is highly effective for large-scale video analysis. Future work will focus on more advanced action detection schemes and better modeling of spatial-temporal relationships to improve performance on subtle activities.
Summary updated