You Reap What You Sow: Using Videos to Generate High Precision Object Proposals for Weakly-Supervised Object Detection

Krishna Kumar Singh, Yong Jae LeeCVPR 2019 · OpenReview

Abstract

We propose a novel way of using videos to obtain high precision object proposals for weakly-supervised object detection. Existing weakly-supervised detection approaches use off-the-shelf proposal methods like edge boxes or selective search to obtain candidate boxes. These methods provide high recall but at the expense of thousands of noisy proposals. Thus, the entire burden of finding the few relevant object regions is left to the ensuing object mining step. To mitigate this issue, we focus instead on improving the precision of the initial candidate object proposals. Since we cannot rely on localization annotations, we turn to video and leverage motion cues to automatically estimate the extent of objects to train a Weakly-supervised Region Proposal Network (W-RPN). We use the W-RPN to generate high precision object proposals, which are in turn used to re-rank high recall proposals like edge boxes or selective search according to their spatial overlap. Our W-RPN proposals lead to significant improvement in performance for state-of-the-art weakly-supervised object detection approaches on PASCAL VOC 2007 and 2012.

Introduction

Object detection typically requires thousands of expensive, manually-drawn bounding box annotations. Weakly-supervised object detection (WSOD) attempts to bypass this by training models using only image-level labels (e.g., "this image contains a car" but not "the car is at these coordinates").

The standard pipeline for WSOD starts by generating thousands of "candidate boxes" using off-the-shelf methods like Selective Search or Edge Boxes. This creates a "needle in a haystack" problem: the model must find the one or two correct boxes among 2,000+ noisy candidates without ever seeing a ground-truth box. Consequently, models often get stuck in local optima, focusing only on the most "distinctive" part of an object (like a bird's beak) or including bits of the background that always appear with the object (like a boat in water).

Methodology: Leveraging Motion for Precision

The authors propose a three-stage approach to move from "high recall/low precision" proposals to "high precision" ones by leveraging unlabeled video data.

1. Learning from Motion

In videos, objects that move differently from their background provide a "free" supervisory signal. The authors use unsupervised motion segmentation (Non-Local Consensus Voting) on large video datasets to identify moving objects. They then treat the tightest boxes around these segments as "pseudo ground-truth."

2. The Weakly-supervised Region Proposal Network (W-RPN)

A Region Proposal Network (W-RPN) is trained on these pseudo-labels. To ensure quality, an image classifier selects video frames where the object is most salient, and an "outlier score" filters out noisy or blurry motion masks. This ensures the network learns from the most reliable examples.

3. High-Precision Proposals & Ranking

When applied to static images, the W-RPN generates a small set (~10) of very reliable proposals. These are combined with Class Activation Maps (CAMs) to ensure they overlap with the relevant object class. Finally, these high-precision boxes are used to re-rank the thousands of noisy standard proposals using a novel ranking loss, forcing the detector to prefer boxes that align with the W-RPN's predictions.

The proposed framework: (a) Training the W-RPN using motion cues from video, and (b) Using W-RPN proposals to train a weakly-supervised object detector.

Key Insights and Results

Precision over Recall

Traditional proposal methods prioritize recall (making sure the object is in one of the 2,000 boxes). This paper argues that for weak supervision, precision is more important. Having 10 "mostly right" boxes is better for a weak learner than 2,000 "mostly wrong" ones.

Comparison of proposal quality. Standard methods (middle) produce thousands of noisy candidates, while the W-RPN (right) generates a small set of high-precision proposals.

Motion as a Proxy for Extent

One of the biggest failures in WSOD is "part-based detection" (detecting just a face instead of a body). Because an entire object usually moves as a single unit in a video, motion cues naturally define the full extent of the object, providing the spatial information that image-level labels lack.

Performance Benchmarks

The method was evaluated on the PASCAL VOC 2007 and 2012 benchmarks, using video data from YouTube-Bounding Boxes and YFCC100m for training.

  • Performance Boost: Adding W-RPN proposals to the standard WSDDN (Weakly Supervised Deep Detection Network) increased Mean Average Precision (mAP) on PASCAL VOC 2007 from 34.8% to 39.4%.
  • State-of-the-Art: When combined with more recent architectures like OICR, the method achieved 46.9% mAP, outperforming previous methods that relied on complex "track and transfer" schemes.

Conclusion

By shifting the focus from high-recall to high-precision proposals, this work demonstrates that unlabeled video can provide a powerful supervisory signal for object detection. The W-RPN successfully captures the full extent of objects, mitigating the common WSOD issue of focusing on discriminative parts. While the method relies on motion, camera movement often provides sufficient cues even for stationary objects, making it a robust approach for improving weakly-supervised learning.

Summary updated

Abstract

We propose a novel way of using videos to obtain high precision object proposals for weakly-supervised object detection. Existing weakly-supervised detection approaches use off-the-shelf proposal methods like edge boxes or selective search to obtain candidate boxes. These methods provide high recall but at the expense of thousands of noisy proposals. Thus, the entire burden of finding the few relevant object regions is left to the ensuing object mining step. To mitigate this issue, we focus instead on improving the precision of the initial candidate object proposals. Since we cannot rely on localization annotations, we turn to video and leverage motion cues to automatically estimate the extent of objects to train a Weakly-supervised Region Proposal Network (W-RPN). We use the W-RPN to generate high precision object proposals, which are in turn used to re-rank high recall proposals like edge boxes or selective search according to their spatial overlap. Our W-RPN proposals lead to significant improvement in performance for state-of-the-art weakly-supervised object detection approaches on PASCAL VOC 2007 and 2012.
BibTeX
@inproceedings{singh2019you,
  author = {Krishna Kumar Singh and Yong Jae Lee},
  title = {{You Reap What You Sow: Using Videos to Generate High Precision Object Proposals for Weakly-Supervised Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=rG6TFBIaWNx},
}
Figure