Dual Bipartite Graph Learning: A General Approach for Domain Adaptive Object Detection

Chaoqi Chen, Jiongcheng Li, Zebiao Zheng, Yue Huang, Xinghao Ding, Yizhou YuICCV 2021 · OpenReview

Abstract

Domain Adaptive Object Detection (DAOD) relieves the reliance on large-scale annotated data by transferring the knowledge learned from a labeled source domain to a new unlabeled target domain. Recent DAOD approaches resort to local feature alignment in virtue of domain adversarial training in conjunction with the ad-hoc detection pipelines to achieve feature adaptation. However, these methods are limited to adapt the specific types of object detectors and do not explore the cross-domain topological relations. In this paper, we first formulate DAOD as an open-set domain adaptation problem in which foregrounds (pixel or region) can be seen as the “known class”, while backgrounds (pixel or region) are referred to as the “unknown class”. To this end, we present a new and general perspective for DAOD named Dual Bipartite Graph Learning (DBGL), which captures the cross-domain interactions on both pixel-level and semantic-level via increasing the distinction between foregrounds and backgrounds and modeling the cross-domain dependencies among different semantic categories. Experiments reveal that the proposed DBGL in conjunction with one-stage and two-stage detectors exceeds the state-of-the-art performance on standard DAOD benchmarks.

Introduction

Deep learning models for object detection often suffer from performance degradation when deployed in environments that differ from their training data—a phenomenon known as domain shift. For instance, a model trained on real-world photographs (source domain) may fail to accurately detect objects in artistic sketches or comic books (target domain).

Current Domain Adaptive Object Detection (DAOD) methods face two primary challenges. First, many are "ad-hoc," designed specifically for two-stage architectures like Faster R-CNN, making them difficult to generalize to one-stage detectors like SSD. Second, standard alignment techniques often attempt to align the entire image distribution. While foreground objects like "dogs" share some semantic consistency across domains, "background" elements (e.g., textures, roads, or abstract patterns) vary significantly. Forcing the alignment of these non-transferable backgrounds often leads to "negative transfer," where the model's performance actually decreases.

Conceptual overview of DAOD as an open-set problem, distinguishing foreground (known) from background (unknown) classes.

Methodology: Dual Bipartite Graph Learning (DBGL)

To address these issues, we propose Dual Bipartite Graph Learning (DBGL). This framework re-conceptualizes DAOD as an Open-Set Domain Adaptation (OSDA) problem. In this view, foreground pixels or regions are treated as "known classes" to be aligned, while backgrounds are treated as an "unknown class" to be isolated and ignored.

The DBGL architecture, featuring Pixel-Level Bipartite Graph Learning (PBGL) and Semantic-Level Bipartite Graph Learning (SBGL).

The DBGL framework consists of two hierarchical components:

Pixel-Level Bipartite Graph Learning (PBGL)

Operating on low-level features, PBGL utilizes "pixel prototypes" to identify mutual nearest neighbors between the source and target domains. By performing message passing across a bipartite graph, the model learns to distinguish foreground pixels from background noise in an unsupervised manner, effectively filtering out non-transferable information early in the pipeline.

Semantic-Level Bipartite Graph Learning (SBGL)

SBGL operates on high-level instance features to model topological relationships between different object categories. To mitigate the "hubness problem"—where certain generic features appear as neighbors to many disparate points—we introduce Cross-Domain Similarity Regularization (CDSR). This ensures that only reliable, mutual matches drive the alignment process. Finally, a Category-aware Domain Alignment (CDA) loss contrastively aligns source and target prototypes, ensuring category-specific consistency in the feature space.

Experimental Results

The DBGL framework was evaluated on three cross-domain benchmarks: Pascal VOC to Clipart1k, Watercolor2k, and Comic2k. These datasets represent significant shifts in visual style (texture, color, and lines) while maintaining consistent object categories.

DBGL achieved state-of-the-art performance for both one-stage and two-stage detectors:

  • Two-Stage (Faster R-CNN): On the VOC → Clipart task, DBGL reached 41.6% mAP, a 1.3% improvement over previous benchmarks.
  • One-Stage (SSD): On the same task, it achieved 37.7% mAP, representing a 2.0% increase over prior methods.

Qualitative analysis indicates that DBGL is particularly effective at detecting sample-scarce categories and maintaining robustness under occlusion.

Conclusion and Future Work

DBGL provides a general, detector-agnostic framework for DAOD by treating backgrounds as an unknown class and leveraging bipartite graphs to capture cross-domain topology. While the method is robust, future work could address the computational complexity of graph construction for high-resolution images and the potential impact of noisy pseudo-labels during early training phases. Exploring more complex topological structures beyond bipartite graphs remains a promising direction for further enhancing domain-adaptive reasoning.

Summary updated

Abstract

Domain Adaptive Object Detection (DAOD) relieves the reliance on large-scale annotated data by transferring the knowledge learned from a labeled source domain to a new unlabeled target domain. Recent DAOD approaches resort to local feature alignment in virtue of domain adversarial training in conjunction with the ad-hoc detection pipelines to achieve feature adaptation. However, these methods are limited to adapt the specific types of object detectors and do not explore the cross-domain topological relations. In this paper, we first formulate DAOD as an open-set domain adaptation problem in which foregrounds (pixel or region) can be seen as the “known class”, while backgrounds (pixel or region) are referred to as the “unknown class”. To this end, we present a new and general perspective for DAOD named Dual Bipartite Graph Learning (DBGL), which captures the cross-domain interactions on both pixel-level and semantic-level via increasing the distinction between foregrounds and backgrounds and modeling the cross-domain dependencies among different semantic categories. Experiments reveal that the proposed DBGL in conjunction with one-stage and two-stage detectors exceeds the state-of-the-art performance on standard DAOD benchmarks.
BibTeX
@inproceedings{chen2021dual,
  author = {Chaoqi Chen and Jiongcheng Li and Zebiao Zheng and Yue Huang and Xinghao Ding and Yizhou Yu},
  title = {{Dual Bipartite Graph Learning: A General Approach for Domain Adaptive Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=7zWirT-Xmf},
}
Figure