TL;DR
This paper introduces D2F2WOD, a novel framework that tackles the challenge of Weakly-Supervised Object Detection (WSOD) by leveraging fully-annotated synthetic data to improve object proposal generation in natural image datasets with only image-level labels. By progressively adapting a Fully-Supervised Object Detector (FSOD) from synthetic to real domains, D2F2WOD significantly enhances both the accuracy and localization capabilities of WSOD models.
What Problem Does This Paper Address?
Object detection is a fundamental computer vision task, but training state-of-the-art detectors typically requires vast amounts of meticulously annotated data, including precise bounding boxes for each object. This annotation process is expensive and time-consuming. Weakly-Supervised Object Detection (WSOD) aims to alleviate this burden by training models using only image-level labels (i.e., knowing that an object class is present in an image, but not where). However, this lack of localization information often leads to suboptimal performance, particularly in generating accurate object proposals, which are crucial for subsequent classification and localization. Existing WSOD methods struggle with the domain gap between synthetic datasets (often used for pre-training due to their rich annotations) and real-world target datasets, leading to poor generalization.
Methods
D2F2WOD proposes a two-stage approach:
- Warm-Up Domain Adaptation Stage: This stage focuses on adapting a Fully-Supervised Object Detector (FSOD) from a source domain (synthetic, fully-annotated) to a target domain (natural, weakly-annotated). This is achieved through a progressive domain adaptation process across five steps:
- Initial Bridge: Using unpaired image-to-image translation (CycleGAN) to create "target-like" intermediate images from source images, preserving object localization but altering appearance.
- Copy-Paste Augmentation: Fusing translated object appearances with real background images to create a second intermediate domain, further bridging the domain gap.
- Pseudo-Labeling (PL): Using the adapted FSOD to generate pseudo-labels for the target domain images.
- Iterative Refinement: Repeatedly fine-tuning the FSOD on pseudo-labeled target data, with augmentations, to gradually improve its performance on the target domain. This process aims to create target-domain-specific and detection-aware object proposals.
- Main WSOD Stage: The adapted FSOD from the warm-up stage is then used to initialize a WSOD model. Specifically:
- The feature extractor of the WSOD model is initialized with the fine-tuned FSOD's feature extractor.
- The object proposal generator of the WSOD model is replaced by the entire fine-tuned FSOD model, treating its detection outputs as object proposals.
- This initialized WSOD model is then trained on the target domain's image-level labels.
This approach allows D2F2WOD to leverage the precise localization information from synthetic data to improve object proposals, which are then used by a standard WSOD framework.
Datasets
The paper evaluates D2F2WOD on five dual-domain image benchmark pairs:
- SyntheticPizza10 → RealPizza10: A newly constructed benchmark pair. SyntheticPizza10 is derived from existing synthetic pizza datasets, and RealPizza10 is a subset of the PizzaGAN dataset, both focusing on pizza ingredients. This pair is noted as particularly challenging due to small object sizes and diverse textures.
- Clipart1K → VOC2007
- Watercolor2K → VOC2007-sub
- Comic2K → VOC2007-sub
- Clipart1K → MS-COCO-sub
For the benchmark construction, datasets were filtered to ensure consistent class sets between source and target domains.
Key Insights and Ideas
- Bridging the Domain Gap via Progressive Adaptation: Instead of a single large domain shift, D2F2WOD breaks down the adaptation into smaller, manageable steps using intermediate domains. This gradual approach is crucial for handling significant differences between synthetic and real-world data.
- Leveraging Synthetic Data for Proposal Quality: The core idea is that synthetic data, despite potential appearance differences, can provide accurate object localization. By adapting an FSOD on this data, the framework learns to generate high-quality proposals that are then beneficial for WSOD.
- Detection-Aware Features: Features learned through the FSOD adaptation process are not just general image features (like those from ImageNet pre-training) but are specifically tuned for object detection tasks (classification and localization), making them more effective for the downstream WSOD model.
- Replacing Hand-Crafted Proposals: The framework replaces traditional, hand-crafted object proposal methods (like Selective Search) with proposals generated by the adapted FSOD. This allows for more accurate and domain-specific proposals.
- Generalizability: The proposed framework is designed to be general and can be applied to various FSOD backbones (e.g., Faster R-CNN, Sparse DETR) and WSOD methods (e.g., OICR, CASD).
Key Results
- D2F2WOD consistently outperforms state-of-the-art single-domain (SD) and cross-domain (CD) WSOD baselines across all five tested benchmarks.
- On Clipart1K → VOC2007, D2F2WODcasd achieved 64.8% mAP, outperforming the original CASD by 7.8% and CASD+W2N by 1.5%.
- On the challenging SyntheticPizza10 → RealPizza10 benchmark, D2F2WODcasd provided a 12.2% improvement over the original CASD.
- The paper demonstrates significant gains over CD baselines like DT+PL and PADOD, with improvements of up to 40.6% mAP.
- Qualitative results show D2F2WOD producing more accurate bounding boxes that closely align with objects of interest, compared to imprecise boxes from baselines.
- Ablation studies confirm the effectiveness of both the domain-specific pre-trained deep features and the weakly-supervised object proposal generator.
Limitations and Future Work
- Computational Overhead: The framework requires extra training time for CycleGAN, which adds to the overall computational cost.
- Joint Learning: The current approach separates the domain adaptation and WSOD stages. The authors suggest that an end-to-end framework could potentially offer further improvements by jointly learning and refining all stages, although controlling back-propagation errors from pseudo-labeling would be a challenge.
- Further Improvement: The paper implies that an end-to-end approach could lead to increasingly discriminative object features and better detection performance on the target domain.
Summary updated