Introduction
Object detection typically requires massive datasets of manually annotated bounding boxes, a process that is both time-consuming and expensive. Weakly-supervised object detection (WSOD) attempts to mitigate this by learning from high-level image labels (e.g., "this image contains a dog"). However, extending WSOD to video data introduces significant challenges, including motion blur and the "domain shift" between static photos and dynamic video frames.
This paper proposes a novel framework that leverages high-level action descriptions (e.g., "sitting in a chair") as a source of supervision for video object detection. By utilizing a Temporal Dynamic Graph LSTM (TD-Graph LSTM), the model learns to propagate information across frames, effectively "filling in the blanks" for objects that are present but not explicitly mentioned in a specific action label.
The "Missing Label" Problem in Video
When using action labels for supervision, researchers encounter a significant hurdle known as the "missing label" problem. In a video snippet labeled "tidying a shelf," the shelf is the primary focus, but other objects like a laptop, a chair, or a refrigerator may also be visible. Standard WSOD models often treat these unmentioned objects as background noise, which confuses the detector and degrades performance.
Methods: Temporal Dynamic Graph LSTM
The core innovation of this work is the TD-Graph LSTM, designed to reason about object relationships across the temporal dimension of a video. The framework operates in four primary stages:
- Spatial Feature Extraction: Each video frame is processed through a ConvNet (VGG-CNN-F) to generate region proposals and extract visual features.
- Dynamic Graph Construction: The model constructs a graph where nodes represent object proposals. Edges are dynamically created between proposals in consecutive frames based on visual similarity, allowing the model to track object instances through time.
- TD-Graph LSTM Unit: Unlike standard LSTMs that process data in a fixed sequence, this unit propagates information across the edges of the dynamic graph. To maintain computational efficiency, the model employs shared frame-level hidden and memory states that summarize the temporal context of preceding frames.
- Two-Stream Classification: Building on the Weakly Supervised Deep Detection Network (WSDDN) architecture, the model uses a classification stream to identify object categories and a detection stream to localize the most probable regions.
Experimental Results
The TD-Graph LSTM was evaluated on the Charades dataset, a large-scale collection of daily life activities. To provide a rigorous evaluation, the authors released a new benchmark of 5,000 manually annotated frames for 17 object categories.
Key Performance Metrics
- Classification: The model achieved a mean Average Precision (mAP) of 19.52%, significantly outperforming baseline methods like ContextLocNet (16.47%) and WSDDN (15.67%).
- Detection: In the challenging "slightly-supervised" detection setting, the model reached 1.98% mAP, nearly tripling the performance of WSDDN (0.65%).
- Small Object Detection: The graph-based propagation allowed for substantial improvements in detecting small or frequently unlabelled objects. For instance, detection AP for "cup" improved from 0.03% to 0.49%.
Conclusion
The TD-Graph LSTM demonstrates that temporal coherence is a powerful signal for weakly-supervised learning. By linking regions across time, the model can "transfer" labels from frames where an object is the focus of an action to frames where it is merely present in the background. While absolute detection accuracy remains lower than fully-supervised methods, this work paves the way for utilizing "cheap" supervision—such as action tags and textual descriptions—to scale object detection to vast, unannotated video libraries.
Summary updated