PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation

Zelin Zang, Fei Wang, Liangyu Li, Jinlin Wu, Chunshui Zhao, Zhen Lei, Baigui SunarXiv 2025 · DOI · arXiv

Introduction

Unsupervised Domain Adaptation (UDA) is critical for deploying models trained on synthetic data to real-world scenarios. While Vision Transformers (ViTs) have become a standard architecture for these tasks, they often struggle with the Foreground Object Mismatch (FOM) Issue. This problem arises when foreground objects vary significantly in scale and position between domains, leading cross-attention mechanisms to erroneously align background noise instead of the objects of interest.

Illustration of the Foreground Object Mismatch (FOM) issue, where significant differences in foreground ratios between source and target domains lead to performance drops.

The PCaM Framework

The authors propose PCaM (Progressive Focus Cross-Attention Mechanism), a lightweight, plug-and-play module designed to harmonize foreground features. The framework operates in three stages:

  1. Attention Rollout (AR): This stage identifies critical image patches by propagating attention weights from the final layers back to the input, creating a reliable foreground map.
  2. Feature Refinement (FR): Based on the AR maps, the model crops and resizes foreground features to a consistent scale, ensuring "apples-to-apples" comparisons across domains regardless of original object size.
  3. Progressively Focused Loss (PF Loss): This loss function encourages the model to focus on localized, continuous regions near the object's center, preventing attention from drifting into the background as training progresses.

Experimental Results

PCaM was evaluated on four major benchmarks: Office-Home, VisDA-2017, DomainNet, and a Remote Sensing dataset (AID→NWPU). The method consistently outperformed state-of-the-art (SOTA) baselines, particularly in categories most affected by scale discrepancies.

Accuracy comparison between the baseline CDTrans and the proposed PCaM, demonstrating significant gains in categories prone to foreground mismatch.

On VisDA-2017, PCaM achieved 91.4% accuracy, a 3.0% improvement over the CDTrans baseline. On DomainNet, the largest and most challenging UDA benchmark, it reached 47.2% average accuracy. The model showed particularly strong gains in categories highly susceptible to FOM; for instance, accuracy for the "truck" category in VisDA jumped from 55.3% to 63.8%.

While PCaM adds a slight computational overhead—increasing training time by approximately 21 seconds per 10 epochs on VisDA—the substantial accuracy improvements justify the trade-off for most practical applications.

Conclusion

PCaM addresses a fundamental weakness in ViT-based domain adaptation by using attention as a geometric guide for spatial normalization. By progressively focusing on foreground objects, it achieves robust feature alignment and sets new performance benchmarks across diverse datasets. Future work may extend this mechanism to multi-modal and video domain adaptation tasks where spatial and temporal mismatches are prevalent.

Summary updated

Abstract

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Recent UDA methods based on Vision Transformers (ViTs) have achieved strong performance through attention-based feature alignment. However, we identify a key limitation: foreground object mismatch, where the discrepancy in foreground object size and spatial distribution across domains weakens attention consistency and hampers effective domain alignment. To address this issue, we propose the Progressive Focus Cross-Attention Mechanism (PCaM), which progressively filters out background information during cross-attention, allowing the model to focus on and fuse discriminative foreground semantics across domains. We further introduce an attentional guidance loss that explicitly directs attention toward task-relevant regions, enhancing cross-domain attention consistency. PCaM is lightweight, architecture-agnostic, and easy to integrate into existing ViT-based UDA pipelines. Extensive experiments on Office-Home, DomainNet, VisDA-2017, and remote sensing datasets demonstrate that PCaM significantly improves adaptation performance and achieves new state-of-the-art results, validating the effectiveness of attention-guided foreground fusion for domain adaptation.
BibTeX
@misc{zang2025pcam,
  author = {Zelin Zang and Fei Wang and Liangyu Li and Jinlin Wu and Chunshui Zhao and Zhen Lei and Baigui Sun},
  title = {{PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation}},
  year = {2025},
  eprint = {2506.17232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.17232},
}
Figure