Introduction
Unsupervised Domain Adaptation (UDA) is critical for deploying models trained on synthetic data to real-world scenarios. While Vision Transformers (ViTs) have become a standard architecture for these tasks, they often struggle with the Foreground Object Mismatch (FOM) Issue. This problem arises when foreground objects vary significantly in scale and position between domains, leading cross-attention mechanisms to erroneously align background noise instead of the objects of interest.
The PCaM Framework
The authors propose PCaM (Progressive Focus Cross-Attention Mechanism), a lightweight, plug-and-play module designed to harmonize foreground features. The framework operates in three stages:
- Attention Rollout (AR): This stage identifies critical image patches by propagating attention weights from the final layers back to the input, creating a reliable foreground map.
- Feature Refinement (FR): Based on the AR maps, the model crops and resizes foreground features to a consistent scale, ensuring "apples-to-apples" comparisons across domains regardless of original object size.
- Progressively Focused Loss (PF Loss): This loss function encourages the model to focus on localized, continuous regions near the object's center, preventing attention from drifting into the background as training progresses.
Experimental Results
PCaM was evaluated on four major benchmarks: Office-Home, VisDA-2017, DomainNet, and a Remote Sensing dataset (AID→NWPU). The method consistently outperformed state-of-the-art (SOTA) baselines, particularly in categories most affected by scale discrepancies.
On VisDA-2017, PCaM achieved 91.4% accuracy, a 3.0% improvement over the CDTrans baseline. On DomainNet, the largest and most challenging UDA benchmark, it reached 47.2% average accuracy. The model showed particularly strong gains in categories highly susceptible to FOM; for instance, accuracy for the "truck" category in VisDA jumped from 55.3% to 63.8%.
While PCaM adds a slight computational overhead—increasing training time by approximately 21 seconds per 10 epochs on VisDA—the substantial accuracy improvements justify the trade-off for most practical applications.
Conclusion
PCaM addresses a fundamental weakness in ViT-based domain adaptation by using attention as a geometric guide for spatial normalization. By progressively focusing on foreground objects, it achieves robust feature alignment and sets new performance benchmarks across diverse datasets. Future work may extend this mechanism to multi-modal and video domain adaptation tasks where spatial and temporal mismatches are prevalent.
Summary updated