Informative and Consistent Correspondence Mining for Cross-Domain Weakly Supervised Object Detection

Luwei Hou, Yu Zhang, Kui Fu, Jia LiCVPR 2021 · OpenReview

Abstract

Cross-domain weakly supervised object detection aims to adapt object-level knowledge from a fully labeled source domain dataset to train object detectors for target domains that are weakly labeled with image-level tags. Instead of domain-level distribution matching, we propose to learn pixel-wise cross-domain correspondences for more precise knowledge transfer. This is realized through a novel cross-domain co-attention scheme trained as region competition. In this scheme, the cross-domain correspondence module seeks informative features on the target domain image, while a collaborative mask generator competes to mask out relevant regions. This competitive learning reveals the accurate object extent in the target domain. To alleviate ambiguity, a domain-cycle consistency regularizer leverages reliable intra-domain correspondence. Our approach achieves consistent improvements over existing methods across various datasets.

Introduction

Training robust object detectors typically requires extensive datasets with precise bounding box annotations, which are costly to obtain. Domain adaptation offers a solution by transferring knowledge from a labeled "source" domain (e.g., real-world photos) to a "target" domain (e.g., artistic styles). In the Weakly Supervised Object Detection (WSOD) setting, the target domain lacks bounding boxes and only provides image-level tags.

Existing methods often rely on global feature alignment to bridge the domain gap. However, global alignment frequently fails to capture the fine-grained, object-level details necessary for accurate localization. Furthermore, weakly supervised models often suffer from "discriminative part" bias, where the detector focuses only on the most salient part of an object (e.g., a bird's head) rather than its entire body.

Methodology

We propose a framework that shifts the focus from global alignment to learning local, pixel-wise correspondences between domains. The architecture is built upon a Faster R-CNN backbone and introduces two core modules: Informative Correspondence Mining (ICM) and Consistent Correspondence Mining (CCM).

Overview of the proposed framework featuring Informative Correspondence Mining (ICM) and Consistent Correspondence Mining (CCM).

Informative Correspondence Mining (ICM)

ICM establishes pixel-to-pixel links between source and target images using a co-attention mechanism. To address the discriminative part bias, we introduce a Region Competition strategy. An adversarial mask generator competes with the correspondence module by hiding the most informative regions of the target image. This forces the model to identify all relevant pixels that characterize the object's full extent, leading to more complete and accurate bounding boxes.

Consistent Correspondence Mining (CCM)

Direct cross-domain matching is inherently noisy due to significant style discrepancies (e.g., matching a photorealistic horse to a stylized clipart version). CCM addresses this by employing Cycle Consistency. It utilizes intra-domain matching within the target domain, which is more reliable as it avoids the style gap. By ensuring that a cycle of transformations (Target A → Source → Target B) matches the direct intra-domain path (Target A → Target B), the model receives stable guidance for the difficult cross-domain mapping.

Experimental Results

The proposed method was evaluated on three challenging target domains: Clipart1k, Watercolor2k, and Comic2k, using Pascal-VOC as the source domain.

Quantitative Performance

Our approach demonstrates substantial improvements in mean Average Precision (mAP) over state-of-the-art methods:

  • Clipart1k: Achieved 46.7% mAP, a 10.4% increase over the previous best (CDWSDA).
  • Watercolor2k: Reached 57.4% mAP, surpassing the baseline of 47.8%.
  • Comic2k: Reached 37.1% mAP, up from 32.9%.

Qualitative Analysis

Visual comparisons highlight the model's superior ability to handle cluttered scenes and localize multiple objects accurately. Unlike baseline methods that often produce loose or incomplete bounding boxes, our method aligns closely with ground truth annotations.

Qualitative comparison of detection results across different methods on various target domains.

Conclusion

This work demonstrates that local, pixel-wise correspondence is more effective than global alignment for cross-domain weakly supervised object detection. By combining region competition with cycle-consistent guidance, the model successfully captures full object geometries across domains. Future research will explore sampling strategies to better handle class imbalances in target datasets.

Summary updated

Abstract

Cross-domain weakly supervised object detection aims to adapt object-level knowledge from a fully labeled source domain dataset (i.e. with object bounding boxes) to train object detectors for target domains that are weakly labeled (i.e. with image-level tags). Instead of domain-level distribution matching, as popularly adopted in the literature, we propose to learn pixel-wise cross-domain correspondences for more precise knowledge transfer. It is realized through a novel cross-domain co-attention scheme trained as region competition. In this scheme, the cross-domain correspondence module seeks for informative features on the target domain image, which after being warped to the source domain image, could best explain its annotations. Meanwhile, a collaborative mask generator competes to mask out the relevant target image region to make the remaining features uninformative. Such competitive learning strives to correlate the full foreground in cross-domain image pairs, revealing the accurate object extent in target domain. To alleviate the ambiguity of inter-domain correspondence learning, a domain-cycle consistency regularizer is futher proposed to leverage the more reliable intra-domain correspondence. The proposed approach achieves consistent improvements over existing approaches by a considerable margin, demonstrated by the experiments on various datasets.
BibTeX
@inproceedings{hou2021informative,
  author = {Luwei Hou and Yu Zhang and Kui Fu and Jia Li},
  title = {{Informative and Consistent Correspondence Mining for Cross-Domain Weakly Supervised Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=vgD6g2skrp},
}
Figure