Abstract
Cross-domain weakly supervised object detection (CDWSOD) aims to adapt a detector from a fully labeled source domain to a weakly labeled target domain. While traditional methods rely on Convolutional Neural Networks (CNNs), they often struggle to capture the global context necessary for effective weak supervision. This paper introduces DETR-GA, the first transformer-based approach for CDWSOD. By incorporating specialized global aggregation modules—specifically class queries in the encoder and a foreground query in the decoder—DETR-GA leverages the inherent attention mechanisms of the DEtection TRansformer (DETR) to bridge the domain gap. Experimental results demonstrate that DETR-GA achieves state-of-the-art performance across multiple benchmarks, significantly outperforming existing CNN-based frameworks.
1. Introduction
Object detection models frequently experience performance degradation when deployed in environments that differ from their training data, a phenomenon known as domain shift. For instance, a model trained on photorealistic images (e.g., PASCAL VOC) often fails when applied to artistic styles like clipart or comics. Cross-domain weakly supervised object detection (CDWSOD) addresses this by utilizing "strong" labels (bounding boxes) from a source domain and "weak" labels (image-level tags) from a target domain.
Previous CDWSOD attempts primarily utilized CNN architectures. However, CNNs are inherently local operators, requiring additional mechanisms like global average pooling to aggregate image-wide information for weak supervision. This paper argues that the DETR architecture is a more natural fit for CDWSOD because its global self-attention mechanism can inherently relate distant pixels, facilitating better semantic understanding without the need for manual pooling tricks.
2. Methodology
The proposed DETR-GA (Global Aggregation) framework modifies the standard DETR architecture to better handle the dual requirements of localization (from the source) and classification (from the target).
2.1 Encoder with Class Queries
In a standard DETR, the encoder processes feature tokens derived from a backbone. DETR-GA introduces a set of learnable class queries into the encoder. Each query corresponds to a specific object category. Through cross-attention, these queries "probe" the entire image to identify relevant semantic features. This allows the encoder to generate image-level predictions that can be directly supervised by the weak labels available in the target domain, effectively turning the encoder into a global semantic aggregator.
2.2 Decoder with a Foreground Query
The decoder is enhanced with a specialized foreground query. Unlike standard object queries that utilize position embeddings to focus on specific spatial regions, the foreground query is assigned a null position embedding. This lack of spatial bias forces the query to aggregate information globally across all potential foreground regions.
2.3 Mutual Benefit and Consensus
Because the foreground query and standard object queries share content embeddings and processing layers, they develop a shared representation of object classes. This synergy allows the strong supervision from the source domain (which teaches the model where and what objects are) and the weak supervision from the target domain (which teaches the model what objects look like in the new style) to reinforce one another.
3. Experiments and Results
3.1 Experimental Setup
The researchers evaluated DETR-GA using PASCAL VOC as the source domain and three distinct target domains representing significant visual shifts:
- Clipart (Clipart$_{all}$ and Clipart$_{test}$)
- Watercolor
- Comic
3.2 Performance Metrics
DETR-GA established new state-of-the-art (SOTA) benchmarks across all tested datasets, often by substantial margins:
- PASCAL VOC → Clipart$_{all}$: The model improved mAP from a baseline of 29.0% to 79.4%, significantly exceeding the previous CNN-based SOTA ($H^2FA$) of 69.8%.
- PASCAL VOC → Watercolor: Achieved 60.0% mAP, surpassing the previous best of 59.9%.
- PASCAL VOC → Comic: Achieved 48.5% mAP, compared to the previous best of 46.4%.
3.3 Key Insights
Ablation studies confirmed that the removal of position embeddings for the foreground query was critical. Without position bias, the query is forced to focus on semantic content rather than spatial coordinates. Furthermore, the authors observed a "DETR Paradox": while a vanilla DETR performs similarly to a standard CNN when directly applied to a new domain, its superior potential is only unlocked when adapted via the proposed global aggregation modules.
4. Conclusion and Future Work
DETR-GA demonstrates that the transformer's attention mechanism is highly effective for cross-domain weak supervision. By replacing local pooling with learnable, query-based global aggregation, the model achieves cleaner signals for domain alignment and superior detection accuracy.
Future research will focus on adapting this method for Deformable DETR, which uses sparse attention and may require new strategies to maintain a global "view." Additionally, the authors aim to optimize the computational efficiency of class queries for datasets containing thousands of categories.
Summary updated