DETR with Additional Global Aggregation for Cross-domain Weakly Supervised Object Detection

Zongheng Tang, Yifan Sun, Si Liu, Yi Yang

Abstract

Cross-domain weakly supervised object detection (CDWSOD) aims to adapt a detector from a fully labeled source domain to a weakly labeled target domain. While traditional methods rely on Convolutional Neural Networks (CNNs), they often struggle to capture the global context necessary for effective weak supervision. This paper introduces DETR-GA, the first transformer-based approach for CDWSOD. By incorporating specialized global aggregation modules—specifically class queries in the encoder and a foreground query in the decoder—DETR-GA leverages the inherent attention mechanisms of the DEtection TRansformer (DETR) to bridge the domain gap. Experimental results demonstrate that DETR-GA achieves state-of-the-art performance across multiple benchmarks, significantly outperforming existing CNN-based frameworks.

1. Introduction

Object detection models frequently experience performance degradation when deployed in environments that differ from their training data, a phenomenon known as domain shift. For instance, a model trained on photorealistic images (e.g., PASCAL VOC) often fails when applied to artistic styles like clipart or comics. Cross-domain weakly supervised object detection (CDWSOD) addresses this by utilizing "strong" labels (bounding boxes) from a source domain and "weak" labels (image-level tags) from a target domain.

Previous CDWSOD attempts primarily utilized CNN architectures. However, CNNs are inherently local operators, requiring additional mechanisms like global average pooling to aggregate image-wide information for weak supervision. This paper argues that the DETR architecture is a more natural fit for CDWSOD because its global self-attention mechanism can inherently relate distant pixels, facilitating better semantic understanding without the need for manual pooling tricks.

2. Methodology

The proposed DETR-GA (Global Aggregation) framework modifies the standard DETR architecture to better handle the dual requirements of localization (from the source) and classification (from the target).

2.1 Encoder with Class Queries

In a standard DETR, the encoder processes feature tokens derived from a backbone. DETR-GA introduces a set of learnable class queries into the encoder. Each query corresponds to a specific object category. Through cross-attention, these queries "probe" the entire image to identify relevant semantic features. This allows the encoder to generate image-level predictions that can be directly supervised by the weak labels available in the target domain, effectively turning the encoder into a global semantic aggregator.

2.2 Decoder with a Foreground Query

The decoder is enhanced with a specialized foreground query. Unlike standard object queries that utilize position embeddings to focus on specific spatial regions, the foreground query is assigned a null position embedding. This lack of spatial bias forces the query to aggregate information globally across all potential foreground regions.

2.3 Mutual Benefit and Consensus

Because the foreground query and standard object queries share content embeddings and processing layers, they develop a shared representation of object classes. This synergy allows the strong supervision from the source domain (which teaches the model where and what objects are) and the weak supervision from the target domain (which teaches the model what objects look like in the new style) to reinforce one another.

3. Experiments and Results

3.1 Experimental Setup

The researchers evaluated DETR-GA using PASCAL VOC as the source domain and three distinct target domains representing significant visual shifts:

  • Clipart (Clipart$_{all}$ and Clipart$_{test}$)
  • Watercolor
  • Comic

3.2 Performance Metrics

DETR-GA established new state-of-the-art (SOTA) benchmarks across all tested datasets, often by substantial margins:

  • PASCAL VOC → Clipart$_{all}$: The model improved mAP from a baseline of 29.0% to 79.4%, significantly exceeding the previous CNN-based SOTA ($H^2FA$) of 69.8%.
  • PASCAL VOC → Watercolor: Achieved 60.0% mAP, surpassing the previous best of 59.9%.
  • PASCAL VOC → Comic: Achieved 48.5% mAP, compared to the previous best of 46.4%.

3.3 Key Insights

Ablation studies confirmed that the removal of position embeddings for the foreground query was critical. Without position bias, the query is forced to focus on semantic content rather than spatial coordinates. Furthermore, the authors observed a "DETR Paradox": while a vanilla DETR performs similarly to a standard CNN when directly applied to a new domain, its superior potential is only unlocked when adapted via the proposed global aggregation modules.

4. Conclusion and Future Work

DETR-GA demonstrates that the transformer's attention mechanism is highly effective for cross-domain weak supervision. By replacing local pooling with learnable, query-based global aggregation, the model achieves cleaner signals for domain alignment and superior detection accuracy.

Future research will focus on adapting this method for Deformable DETR, which uses sparse attention and may require new strategies to maintain a global "view." Additionally, the authors aim to optimize the computational efficiency of class queries for datasets containing thousands of categories.

Summary updated

Abstract

This paper presents a DETR-based method for cross-domain weakly supervised object detection (CDWSOD), aiming at adapting the detector from source to target domain through weak supervision. We think DETR has strong potential for CDWSOD due to an insight: the encoder and the decoder in DETR are both based on the attention mechanism and are thus capable of aggregating semantics across the entire image. The aggregation results, i.e., image-level predictions, can naturally exploit the weak supervision for domain alignment. Such motivated, we propose DETR with additional Global Aggregation (DETR-GA), a CDWSOD detector that simultaneously makes "instance-level + image-level" predictions and utilizes "strong + weak" supervisions. The key point of DETR-GA is very simple: for the encoder / decoder, we respectively add multiple class queries / a foreground query to aggregate the semantics into image-level predictions. Our query-based aggregation has two advantages. First, in the encoder, the weakly-supervised class queries are capable of roughly locating the corresponding positions and excluding the distraction from non-relevant regions. Second, through our design, the object queries and the foreground query in the decoder share consensus on the class semantics, therefore making the strong and weak supervision mutually benefit each other for domain alignment. Extensive experiments on four popular cross-domain benchmarks show that DETR-GA significantly improves CSWSOD and advances the states of the art (e.g., 29.0% --> 79.4% mAP on PASCAL VOC --> Clipart_all dataset).
BibTeX
@inproceedings{tang2023detr,
  author = {Zongheng Tang and Yifan Sun and Si Liu and Yi Yang},
  title = {{DETR with Additional Global Aggregation for Cross-domain Weakly Supervised Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure