Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection

Yuan Yuan, Xiaodan Liang, Xiaolong Wang, Dit-Yan Yeung, Abhinav Gupta

Introduction

Object detection typically requires massive datasets of manually annotated bounding boxes, a process that is both time-consuming and expensive. Weakly-supervised object detection (WSOD) attempts to mitigate this by learning from high-level image labels (e.g., "this image contains a dog"). However, extending WSOD to video data introduces significant challenges, including motion blur and the "domain shift" between static photos and dynamic video frames.

This paper proposes a novel framework that leverages high-level action descriptions (e.g., "sitting in a chair") as a source of supervision for video object detection. By utilizing a Temporal Dynamic Graph LSTM (TD-Graph LSTM), the model learns to propagate information across frames, effectively "filling in the blanks" for objects that are present but not explicitly mentioned in a specific action label.

The "Missing Label" Problem in Video

When using action labels for supervision, researchers encounter a significant hurdle known as the "missing label" problem. In a video snippet labeled "tidying a shelf," the shelf is the primary focus, but other objects like a laptop, a chair, or a refrigerator may also be visible. Standard WSOD models often treat these unmentioned objects as background noise, which confuses the detector and degrades performance.

Comparison of standard weakly-supervised detection versus the proposed action-driven video detection task.

Methods: Temporal Dynamic Graph LSTM

The core innovation of this work is the TD-Graph LSTM, designed to reason about object relationships across the temporal dimension of a video. The framework operates in four primary stages:

  1. Spatial Feature Extraction: Each video frame is processed through a ConvNet (VGG-CNN-F) to generate region proposals and extract visual features.
  2. Dynamic Graph Construction: The model constructs a graph where nodes represent object proposals. Edges are dynamically created between proposals in consecutive frames based on visual similarity, allowing the model to track object instances through time.
  3. TD-Graph LSTM Unit: Unlike standard LSTMs that process data in a fixed sequence, this unit propagates information across the edges of the dynamic graph. To maintain computational efficiency, the model employs shared frame-level hidden and memory states that summarize the temporal context of preceding frames.
  4. Two-Stream Classification: Building on the Weakly Supervised Deep Detection Network (WSDDN) architecture, the model uses a classification stream to identify object categories and a detection stream to localize the most probable regions.
Detailed view of action-driven video object detection, showing how action labels correspond to specific frames and the resulting object lists.

Experimental Results

The TD-Graph LSTM was evaluated on the Charades dataset, a large-scale collection of daily life activities. To provide a rigorous evaluation, the authors released a new benchmark of 5,000 manually annotated frames for 17 object categories.

Key Performance Metrics

  • Classification: The model achieved a mean Average Precision (mAP) of 19.52%, significantly outperforming baseline methods like ContextLocNet (16.47%) and WSDDN (15.67%).
  • Detection: In the challenging "slightly-supervised" detection setting, the model reached 1.98% mAP, nearly tripling the performance of WSDDN (0.65%).
  • Small Object Detection: The graph-based propagation allowed for substantial improvements in detecting small or frequently unlabelled objects. For instance, detection AP for "cup" improved from 0.03% to 0.49%.

Conclusion

The TD-Graph LSTM demonstrates that temporal coherence is a powerful signal for weakly-supervised learning. By linking regions across time, the model can "transfer" labels from frames where an object is the focus of an action to frames where it is merely present in the background. While absolute detection accuracy remains lower than fully-supervised methods, this work paves the way for utilizing "cheap" supervision—such as action tags and textual descriptions—to scale object detection to vast, unannotated video libraries.

Summary updated

Abstract

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.
BibTeX
@inproceedings{yuan2017temporal,
  author = {Yuan Yuan and Xiaodan Liang and Xiaolong Wang and Dit-Yan Yeung and Abhinav Gupta},
  title = {{Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.200},
}
Figure