TL;DR
This paper introduces TITAN, a novel framework for source-free domain adaptive object detection (SF-DAOD). TITAN tackles the challenge of adapting object detectors to new domains without access to source data by intelligently partitioning target data into "easy" and "hard" subsets based on prediction variance and employing query-token adversarial learning to bridge domain gaps.
What Problem Does This Paper Address?
Deep learning models, particularly for object detection, often struggle to generalize to new visual domains due to domain shifts. While Unsupervised Domain Adaptation (UDA) aims to address this, it typically requires access to source data during adaptation, which is often impractical due to privacy or logistical constraints. Source-Free Domain Adaptive Object Detection (SF-DAOD) addresses this by adapting a pre-trained model to a new domain using only unlabeled target data. However, existing SF-DAOD methods, often relying on student-teacher frameworks with pseudo-labeling, suffer from training instability. This instability arises from noisy pseudo-labels caused by domain bias and significant domain shifts, which can degrade the teacher model and propagate errors. The paper aims to provide a more robust and effective SF-DAOD solution that mitigates these issues.
Methods
TITAN introduces several key innovations:
- Target Domain Division: The core idea is to leverage the pre-trained model's prediction variance on target images as a proxy for similarity to the source domain. Higher variance indicates greater uncertainty and thus a sample that is more dissimilar to the source (hard sample), while lower variance suggests similarity (easy sample). This allows the target dataset to be split into two subsets.
- Query-Token Based Domain Adversarial Learning: This is the central adversarial component.
- Encoder Query: A special "query" embedding is introduced and concatenated with token sequences in the encoder. This query adaptively aggregates domain-specific features, capturing global context and focusing on tokens with larger domain discrepancies. This aggregated feature is then passed through a domain discriminator (
Denca) to align encoder features. - Decoder Query: Similarly, a query is appended to object queries in the decoder. This query combines contextual information from object queries, capturing relationships among objects. This is then processed by a domain discriminator (
Ddech) for decoder feature alignment. - Token-wise Domain Adversarial Learning: In addition to the global query-based alignment, token-wise adversarial learning is applied to both encoder and decoder. This helps address domain shifts related to local textures and styles, complementing the global alignment.
- Cascaded Feature Alignment: The alignment process is cascaded, progressively aligning features from shallow to deep layers in both the encoder and decoder. This is achieved by combining the query-based and token-based adversarial losses with hyperparameters (
denca,Adecq) to balance their contributions.
The overall objective function minimizes the student model's loss while maximizing the domain discriminators' loss, effectively pushing the student model to produce domain-invariant features.
Datasets
The paper evaluates TITAN on a variety of datasets:
- Natural Image Datasets:
- Cityscapes (CS)
- Foggy Cityscapes (FC)
- KITTI
- SIM10k (synthetic)
- BDD100k
- Medical Imaging Datasets:
- RSNA-BSD1K (Breast Cancer Detection)
- INBreast (Breast Cancer Detection)
- DDSM (Breast Cancer Detection)
The experiments involve cross-domain adaptation scenarios, such as Cityscapes to Foggy Cityscapes (C2F), Cityscapes to BDD100k (C2B), KITTI to Cityscapes (K2C), and SIM10k to Cityscapes (S2C). For medical imaging, they adapt from RSNA-BSD1K to INBreast (R2In) and DDSM to INBreast (D2In).
Key Insights and Ideas
- Prediction Variance as a Proxy for Domain Similarity: The insight that higher detection variance from a source-pretrained model on target data indicates greater dissimilarity to the source domain is crucial. This simple yet effective criterion allows for a principled way to partition target data without needing source labels.
- Query-Token Adversarial Learning Synergy: The paper highlights that query-based and token-wise adversarial learning are complementary. The former captures global context and scene layout, while the latter focuses on local textures and appearances, leading to more comprehensive domain alignment.
- Importance of Target Data Partitioning: The ablation studies demonstrate that the target division strategy significantly boosts performance, indicating that treating "easy" and "hard" samples differently is key to robust adaptation.
- SF-DAOD's Relevance in Medical Imaging: The paper emphasizes the particular importance of SF-DAOD for medical imaging due to data privacy concerns and the common issue of single-center bias in source datasets.
Key Results
TITAN demonstrates superior performance across various benchmarks:
- Natural Image Datasets: The paper reports significant mAP improvements over current SOTA methods on C2F, C2B, S2C, and K2C benchmarks, with gains of +22.7, +22.2, +21.1, and +3.7 percent respectively. For example, on C2F, TITAN achieves 52.2 mAP, outperforming the previous SOTA at 40.3 mAP.
- Medical Imaging Datasets: TITAN achieves state-of-the-art recall scores on cross-domain mammogram datasets for Breast Cancer Detection (BCD), significantly outperforming previous bests. For instance, it achieves recall scores of 0.78 and 0.51 at 0.3 FPI on RSNA-BSD1K and DDSM datasets, respectively.
- Ablation Studies: The studies confirm the effectiveness of individual components. Query-based domain adversarial learning improves mAP by 40.8, token-wise by 39.7, and the combination with target division (TD) leads to a substantial gain to 51.2 mAP.
Limitations and Future Work
- Limitations: The paper acknowledges that while TITAN significantly improves performance, achieving perfect domain adaptation remains a challenge. The reliance on the pre-trained model's variance might be sensitive to the quality of the initial source model.
- Future Work: The authors suggest that their framework can be extended to other computer vision tasks beyond object detection. Further research could explore more sophisticated methods for target data partitioning or investigate adaptive strategies for the hyperparameters controlling the different loss terms. The complementary nature of query-based and token-wise alignment also opens avenues for exploring novel attention mechanisms.
Summary updated