D2DF2WOD: Learning Object Proposals for Weakly-Supervised Object Detection via Progressive Domain Adaptation

Yuting Wang, Ricardo Guerrero, Vladimir Pavlovic

TL;DR

This paper introduces D2F2WOD, a novel framework that tackles the challenge of Weakly-Supervised Object Detection (WSOD) by leveraging precisely annotated synthetic data to improve object localization in natural images with only image-level labels. By progressively adapting a fully-supervised detector from synthetic to real-world domains, D2F2WOD significantly enhances the quality of object proposals, leading to state-of-the-art performance in challenging scenarios.

What Problem Does This Paper Address?

Object detection models typically require extensive, pixel-level annotations (bounding boxes) for training, which are costly and time-consuming to acquire. Weakly-Supervised Object Detection (WSOD) aims to alleviate this burden by using only image-level labels (e.g., "this image contains a cat"). However, this simplification often results in poor object localization and detection accuracy, especially for complex, real-world images. A key bottleneck is the quality of "object proposals" – initial guesses of where objects might be – which are often inaccurate when generated by standard methods. This paper addresses the significant domain gap between easily annotated synthetic datasets (with precise localization) and challenging natural image datasets (with only image-level labels), aiming to bridge this gap to improve WSOD performance.

Methods

The core of D2F2WOD is a two-stage approach:

  1. Warm-up Domain Adaptation Stage: This stage focuses on improving the object proposal generator. It starts by training a Fully-Supervised Object Detector (FSOD) on a source domain of synthetic images with precise annotations. This FSOD model is then progressively fine-tuned through a series of intermediate domains. These intermediate domains are created using techniques like CycleGAN for image-to-image translation and copy-paste augmentation to gradually adapt the appearance and layout from synthetic to target-like images, while preserving accurate localization. This process results in a domain-specific and detection-aware feature extractor and object proposal generator.
  2. Main WSOD Stage: In this stage, a standard WSOD model (like OICR or CASD) is initialized using the components (feature extractor and proposal generator) learned in the warm-up stage. This initialized WSOD model is then trained on the target domain's image-level labels. By leveraging the high-quality, domain-specific proposals generated from the warm-up stage, the WSOD model can achieve significantly better detection and localization performance.

Datasets

The paper evaluates D2F2WOD on five dual-domain benchmark pairs:

  • SyntheticPizza10 → RealPizza10: This is a newly constructed benchmark, with synthetic pizzas generated from clip-art and real-world pizza images annotated with 13 toppings. It's highlighted as particularly challenging due to small object sizes, diverse shapes, and complex occlusions.
  • Clipart1K → VOC2007
  • Watercolor2K → VOC2007-sub
  • Comic2K → VOC2007-sub
  • Clipart1K → MS-COCO-sub

The synthetic datasets (Clipart1K, Watercolor2K, Comic2K, SyntheticPizza10) provide precise instance-level annotations, while the target datasets (VOC2007, MS-COCO, RealPizza10) only have image-level labels.

Key Insights and Ideas

  • Bridging the Domain Gap with Synthetic Data: The core idea is that precisely annotated synthetic data, despite its potentially different appearance, can be a powerful tool to learn accurate object localization for WSOD in natural images.
  • Progressive Domain Adaptation is Crucial: Simply transferring knowledge from synthetic to real domains is insufficient due to the large domain gap. A gradual, step-by-step adaptation process, involving intermediate domains and augmentations, is key to effectively transferring localization knowledge.
  • Object Proposals are the Bottleneck: The paper strongly emphasizes that the quality of object proposals is a primary limitation in current WSOD methods. By improving these proposals, D2F2WOD achieves substantial gains.
  • Detection-Aware Features: Features learned through the FSOD warm-up stage are not just general image features but are specifically tailored for object detection tasks (classification and localization), making them more effective for WSOD.
  • Generalizability: The proposed framework is shown to be applicable to different WSOD architectures (OICR, CASD) and FSOD backbones (Faster R-CNN, Sparse DETR), demonstrating its broad utility.

Key Results

  • State-of-the-Art Performance: D2F2WOD consistently outperforms existing single-domain (SD) and cross-domain (CD) WSOD baselines across all tested benchmarks.
  • Significant Gains on Challenging Datasets: On the challenging SyntheticPizza10 → RealPizza10 benchmark, D2F2WODcasd achieves a 12.2% mAP improvement over the original CASD.
  • Improved Localization: Error analysis using TIDE shows that D2F2WOD significantly reduces localization errors compared to baselines like DT+PL and CASD.
  • Effective Warm-up Stage: The warm-up stage alone provides substantial improvements, demonstrating the value of the progressive domain adaptation for generating better proposals. For instance, on Clipart1K → VOC2007, D2F2WODwarm-up achieves 37.3% mAP, a significant jump from baselines like DT+PL (22.8%) and PADOD (24.2%).
  • Qualitative Improvements: Visual examples show D2F2WOD producing more accurate and tighter bounding boxes compared to CASD, especially on complex images.

Limitations and Future Work

  • Training Time: The use of CycleGAN for image-to-image translation adds significant computational overhead to the training process.
  • End-to-End Learning: The current framework separates the domain adaptation and WSOD stages. An end-to-end approach that jointly learns and refines all stages could potentially yield further improvements, though controlling error propagation from pseudo-labeling remains a challenge.
  • Data Scarcity: The paper notes that for categories with very few training examples (e.g., bacon, broccoli, pineapple in RealPizza10), even D2F2WOD struggles, suggesting that more data or specialized techniques for low-resource categories might be needed.
  • Transformer-based Detectors: While the framework is general, applying it to transformer-based detectors like Sparse DETR on smaller datasets (like Clipart1K) was challenging due to data requirements, suggesting future work in exploring more data-efficient transformer architectures or increasing synthetic data size.

Summary updated

Abstract

Weakly-supervised object detection (WSOD) models attempt to leverage image-level annotations in lieu of accurate but costly-to-obtain object localization labels. This oftentimes leads to substandard object detection and localization at inference time. To tackle this issue, we propose D2DF2WOD, a Dual-Domain Fully-to-Weakly Supervised Object Detection framework that leverages synthetic data, annotated with precise object localization, to supplement a natural image target domain, where only image-level labels are available. In its warm-up domain adaptation stage, the model learns a fully-supervised object detector (FSOD) to improve the precision of the object proposals in the target domain, and at the same time learns target-domain-specific and detection-aware proposal features. In its main WSOD stage, a WSOD model is specifically tuned to the target domain. The feature extractor and the object proposal generator of the WSOD model are built upon the fine-tuned FSOD model. We test D2DF2WOD on five dual-domain image benchmarks. The results show that our method results in consistently improved object detection and localization compared with state-of-the-art methods.
BibTeX
@inproceedings{wang2023d,
  author = {Yuting Wang and Ricardo Guerrero and Vladimir Pavlovic},
  title = {{D2DF2WOD: Learning Object Proposals for Weakly-Supervised Object Detection via Progressive Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=3gvpVleQMA6},
}
Figure