BUPT-MCPRL at TRECVID 2018: ACtEV and INS

Xinkun Cao, Zhipeng Wang, Peng Li, Xuanlu Xiang, Xinyu Li, Ya Li, Shaoqiang Xu, Liang Zhou, Zhicheng Zhao, Yanyun Zhao, Fei SuTRECVID 2018 · OpenReview

Abstract

The BUPT-MCPRL team secured top-three finishes in the TRECVID 2018 competition by deploying a sophisticated multi-stage pipeline for video activity detection and instance search. Their approach combines standard object detection (Faster R-CNN, YOLO) with specialized temporal models like Temporal Relation Networks (TRN) and 3D-ResNets. By fusing trajectory-based heuristics with deep temporal classification, the system effectively manages the computational complexity of surveillance-style video analysis.

Introduction

Large-scale video analysis in surveillance contexts presents two primary challenges: Activities in Extended Video (ActEV) and Instance Search (INS). ActEV requires detecting and localizing human-vehicle interactions (e.g., "opening a trunk") and person-only activities (e.g., "talking on a phone") within long, unedited footage. This is a "needle in a haystack" problem with high computational costs. INS involves finding a specific person at a specific location across a massive video library, necessitating a simultaneous understanding of identity (who) and scene (where).

Methodology

ActEV Pipeline

The BUPT-MCPRL team implemented a "detect-track-propose-classify" architecture to handle the ActEV task.

Overview of the multi-stage ActEV pipeline, from detection to post-processing.
  1. Detection & Tracking: Pedestrians are detected using Faster R-CNN with a ResNet-50 backbone and Online Hard Example Mining (OHEM), while vehicles are identified via YOLO. Tracking is enhanced through deep Re-Identification (Re-ID) features, which provide greater robustness against occlusions and lighting changes compared to traditional color histograms.
  2. Event Proposals: To reduce computational overhead, the system generates proposals using hand-crafted rules. For instance, human-vehicle interactions are only proposed when the Intersection over Union (IoU) between a person and a vehicle exceeds a specific threshold.
  3. Specialized Classification: Events are classified using models tailored to their nature:
  4. Trajectory Analysis: Used for vehicle turning events by analyzing angle variations.
  5. Temporal Models: Convolutional-De-Convolutional (CDC) networks provide precise temporal localization for "Opening/Closing" events, while Temporal Relation Networks (TRN) capture the logical sequence of actions like "Open_Trunk" vs. "Closing_Trunk."
    • 3D-ResNets: Employed for complex actions such as "Entering/Exiting" and "Talking."

Instance Search (INS) Pipeline

The INS system utilizes a dual-path retrieval strategy:

  • Location Path: Combines local features (RootSIFT, Bag-of-Words) with global features (ResNeXt trained on Places365).
  • Person Path: Uses MTCNN for face detection and cosine distance for matching, supplemented by searching video transcripts for character names.
  • Fusion: The system performs "Rank Fusion," combining results from (Location → Person) and (Person → Location) searches. A Random Forest classifier, trained on TRECVID 2016 data, provides a final re-ranking boost.

Key Insights

A central theme of the BUPT-MCPRL approach is the use of heuristics as a filter. By applying simple physical rules (velocity thresholds, proximity) to generate proposals, the team significantly reduced the search space. This allowed for the targeted application of computationally expensive 3D-CNNs. Furthermore, the team distinguished between "states" (discriminated by pose/context, like "Riding") and "actions" (discriminated by motion patterns, like "Closing"), selecting the optimal tool (image vs. temporal classification) for each.

Results and Discussion

The system performed exceptionally well, earning 3rd place on the ActEV Activity Detection (AD) leaderboard and 2nd place on the Activity and Object Detection (AOD) leaderboard.

DET curves showing the detection performance across various activity categories in the ActEV task.

In the INS task, the team achieved a maximum Mean Average Precision (mAP) of 44.7 for interactive search runs and 38.0 mAP for automatic runs. Despite these successes, the authors noted difficulties with "small movement" events like "Talking" or "Interacts," which are often hard to distinguish from background noise.

Conclusion

The BUPT-MCPRL team demonstrated that a multi-stage pipeline combining robust detection, heuristic-based proposals, and specialized temporal classification is highly effective for large-scale video analysis. Future work will focus on more advanced action detection schemes and better modeling of spatial-temporal relationships to improve performance on subtle activities.

Summary updated

BibTeX
@inproceedings{cao2018bupt,
  author = {Xinkun Cao and Zhipeng Wang and Peng Li and Xuanlu Xiang and Xinyu Li and Ya Li and Shaoqiang Xu and Liang Zhou and Zhicheng Zhao and Yanyun Zhao and Fei Su},
  title = {{BUPT-MCPRL at TRECVID 2018: ACtEV and INS}},
  booktitle = {TRECVID},
  year = {2018},
  url = {https://openreview.net/forum?id=bYqdVUpYyPJ},
}
Figure