Leveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection

Mykhailo Shvets, Wei Liu, Alexander C. BergICCV 2019 · DOI

Introduction

Standard object detectors treat video as a sequence of independent images, often failing when objects are obscured by motion blur, temporary occlusions, or unusual poses. While temporal context can fill these gaps, existing methods typically rely on computationally expensive optical flow, which is limited to short-range dependencies. This paper introduces a lightweight Temporal Relation Module that links object proposals across distant frames using appearance similarity, achieving state-of-the-art accuracy at near-real-time speeds.

Methodology

The proposed system operates on object proposals (RoI features) rather than raw pixels. By identifying similar-looking proposals in past frames, the model remains robust to unpredictable spatial movements and camera shake.

Overview of the Temporal Relation Module architecture linking proposals across frames.

Key Components

  • Appearance-based Alignment: Instead of tracking pixel displacement, the module calculates similarity between RoI features. This spatial agnosticism allows the model to handle high-velocity objects that move significantly between frames.
  • Feature Normalization (FeatNorm): To prevent the attention mechanism from being biased by raw feature magnitudes, unit normalization is applied, ensuring the module measures true Pearson correlation.
  • Graph Supervision: The model is trained with a "GraphLoss" that explicitly encourages high similarity scores for proposals belonging to the same object track across the video.
  • Operational Modes: The system supports a Causal mode for real-time applications (looking only at the past) and a Symmetric mode for offline processing (looking at past and future).

Key Insights

A major finding of this work is that long-range strides are often superior to immediate temporal neighbors. Frames very close in time likely suffer from the same degradations (e.g., the same occlusion event). In contrast, a frame from several seconds ago might offer a much clearer view of the target object.

mAP performance relative to aggregation size (T) and base temporal stride.

Furthermore, by aggregating information at the proposal level, the search space is reduced from millions of pixels to a few hundred regions, significantly boosting inference speed without sacrificing accuracy.

Experimental Results

The model was evaluated on the ImageNet VID benchmark, demonstrating significant improvements over single-frame baselines:

  • Accuracy: Achieved 80.7 mAP in causal mode and 84.1 mAP with a ResNeXt-101 backbone.
  • Speed: The causal model runs at 10 FPS on a Titan X Pascal, a negligible overhead compared to the 14 FPS baseline.
  • Fast Motion: The model showed a +9.3 mAP improvement specifically on fast-moving objects, proving the effectiveness of appearance-based relations over spatial tracking.

Conclusion

By shifting the focus from pixel-level flow to proposal-level relationships, this research provides a robust and efficient framework for video object detection. The approach is particularly effective for high-speed motion and long-range temporal dependencies, offering a scalable solution for real-time video analysis tasks.

Summary updated

BibTeX
@inproceedings{shvets2019leveraging,
  author = {Mykhailo Shvets and Wei Liu and Alexander C. Berg},
  title = {{Leveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00985},
}
Figure