Salvage of Supervision in Weakly Supervised Object Detection

Lin Sui, Chen-Lin Zhang, Jianxin Wu

Abstract

Weakly Supervised Object Detection (WSOD) suffers from a significant performance gap compared to fully supervised methods, largely due to its inability to leverage modern architectural advancements. We present Salvage of Supervision (SoS), a three-stage framework that bridges this gap by iteratively refining pseudo-labels and employing semi-supervised learning. SoS enables the use of modern backbones like ResNet and Feature Pyramid Networks (FPN), achieving state-of-the-art results on PASCAL VOC and MS-COCO while significantly improving inference speed.

Introduction

Object detection typically requires "fully supervised" data, where every object is meticulously outlined with a bounding box. Producing such datasets is labor-intensive and expensive. Weakly Supervised Object Detection (WSOD) attempts to learn from "weak" labels—simple image-level tags indicating the presence of an object without spatial information.

However, WSOD has historically struggled to integrate modern detection advancements like ResNet backbones, FPN, and RoIAlign. These techniques, which have propelled Fully Supervised Object Detection (FSOD) to new heights, often degrade performance in traditional WSOD frameworks. Consequently, WSOD models remain slower and less accurate. This paper introduces the SoS framework to "salvage" supervisory signals from weak labels, allowing WSOD models to benefit from modern FSOD architectures.

Methods

The SoS-WSOD framework operates in three distinct stages to refine supervision iteratively:

  1. Improved WSOD (Stage 1): The process begins with a baseline WSOD model (based on OICR) enhanced with optimized proposal mining rules. By balancing precision and recall during proposal selection, the model ensures a more comprehensive initial set of object candidates.
  2. Pseudo-FSOD (Stage 2): The Stage 1 model generates pseudo-bounding boxes for the training set. To mitigate noise, the authors introduce Pseudo Groundtruth Filtering (PGF), which removes redundant, tiny, or low-confidence boxes. These salvaged labels are then used to train a modern FSOD detector (e.g., Faster R-CNN with ResNet50 and FPN) from scratch.
  3. Semi-Supervised Learning (Stage 3): To address remaining noise in pseudo-labels, the dataset is split into "clean" and "noisy" subsets based on a "small loss" criterion. The noisy subset is treated as unlabeled data within a semi-supervised framework (Unbiased Teacher) to further enhance model robustness.
The SoS-WSOD framework pipeline, illustrating the transition from weak supervision to semi-supervised object detection.

Key Insights

  • The Modernity Gap: Modern FSOD tools like RoIAlign can actually harm traditional WSOD because these models lack the initial strength to handle high-resolution features without better supervision.
  • Filtering vs. Excavation: Rather than complex "excavation" of larger boxes, a robust filtering approach (PGF) proves more effective, especially on complex datasets like MS-COCO.
  • Loss as Quality Proxy: Training loss serves as an effective proxy for pseudo-label quality; low-loss images are likely to have cleaner labels, while high-loss images are better utilized as unlabeled data in a semi-supervised context.

Results

SoS-WSOD significantly outperforms previous state-of-the-art methods across standard benchmarks:

  • PASCAL VOC 2007/2012: Achieved 64.4 mAP₅₀ and 61.9 mAP₅₀ respectively, representing a substantial improvement over previous bests (~56.8).
  • MS-COCO: Reached 32.8 mAP₅₀, nearly doubling the performance of many earlier WSOD methods on this challenging dataset.
  • Efficiency: By utilizing a standard Faster R-CNN for the final stage, inference is approximately 3x faster than traditional WSOD models that rely on external object proposal generators.
Qualitative detection results showing the model's ability to identify multiple object classes with high confidence scores.

Discussion and Limitations

While SoS-WSOD narrows the gap, a performance disparity remains compared to true full supervision (which exceeds 80 mAP on VOC). Current limitations include "part domination" (identifying only a portion of an object, like a head), difficulty distinguishing individual instances in crowded clusters, and occasional missing instances during pseudo-labeling. Future research will focus on more sophisticated data splitting and advanced semi-supervised techniques to further recover lost supervision.

Summary updated

Abstract

Weakly supervised object detection~(WSOD) has recently attracted much attention. However, the lack of bounding-box supervision makes its accuracy much lower than fully supervised object detection (FSOD), and currently modern FSOD techniques cannot be applied to WSOD. To bridge the performance and technical gaps between WSOD and FSOD, this paper proposes a new framework, Salvage of Supervision (SoS), with the key idea being to harness every potentially useful supervisory signal in WSOD: the weak image-level labels, the pseudo-labels, and the power of semi-supervised object detection. This paper proposes new approaches to utilize these weak and noisy signals effectively, and shows that each type of supervisory signal brings in notable improvements, outperforms existing WSOD methods (which mainly use only the weak labels) by large margins. The proposed SoS-WSOD method also has the ability to freely use modern FSOD techniques. SoS-WSOD achieves 64.4 $m\text{AP}_{50}$ on VOC2007, 61.9 $m\text{AP}_{50}$ on VOC2012 and 16.6 $m\text{AP}_{50:95}$ on MS-COCO, and also has fast inference speed. Ablations and visualization further verify the effectiveness of SoS.
BibTeX
@inproceedings{sui2022salvage,
  author = {Lin Sui and Chen-Lin Zhang and Jianxin Wu},
  title = {{Salvage of Supervision in Weakly Supervised Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=82Gm7AaLAw},
}
Figure