Attention-based cross-layer domain alignment for unsupervised domain adaptation

Xu Ma, Junkun Yuan, Yen-Wei Chen, Ruofeng Tong, Lanfen Lin

Introduction: The Semantic Dislocation Problem

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Traditional methods typically align features at the same layer across both domains. However, this assumes that semantic information is represented at identical depths in the neural network for both datasets.

The authors identify a critical "semantic dislocation problem": due to domain shifts, a high-level concept (e.g., "car") might be captured in Layer 3 for the source domain but appear in Layer 2 for the target domain. Forcing alignment between identical layers can lead to "negative transfer," where the model's performance is degraded rather than improved.

Comparison of same-layer alignment (a) vs. the proposed cross-layer semantic alignment (b).

Methodology: Attention-based Cross-layer Domain Alignment (ACDA)

To address semantic dislocation, the researchers proposed Attention-based Cross-layer Domain Alignment (ACDA). This framework dynamically identifies and aligns semantically similar features regardless of their layer depth.

The ACDA architecture, featuring convolution-based projection and a dynamic attention mechanism for cross-layer alignment.

The ACDA framework consists of three core components:

  1. Convolution-based Projection: Since layers at different depths have varying dimensions (height, width, and channel count), small convolutional layers project these features into a shared mathematical space for comparison.
  2. Cross-Layer Semantic Alignment: Instead of a rigid 1-to-1 layer mapping, ACDA evaluates the distance between all possible pairs of layers across the source and target domains.
  3. Dynamic Attention Mechanism: This module calculates semantic similarity across spatial and channel dimensions. It assigns higher weights to layer pairs with high similarity, allowing the model to automatically "discover" the optimal alignment points and solve the dislocation problem.

The final objective function combines standard cross-entropy loss ($\mathcal{L}_{ce}$) for classification with a weighted cross-layer alignment loss ($\mathcal{L}_{ali}$).

Experimental Results

The ACDA method was evaluated on four major UDA benchmarks: Office-31, Office-Home, ImageCLEF-DA, and VisDA-2017.

  • Office-31: Achieved an average accuracy of 90.94%, surpassing established methods like JAN and CDAN.
  • Office-Home: Demonstrated a significant performance boost, improving upon the Joint Adaptation Network (JAN) by approximately 12.2%.
  • VisDA-2017: Reached 86.4% accuracy, outperforming competitive baselines such as RWOT (84.0%) and GSDA (81.5%).

Beyond raw accuracy, training curves indicated that cross-layer alignment provides greater stability and faster convergence compared to traditional same-layer methods.

Conclusion and Future Directions

The primary contribution of this work is the recognition that domain shift affects the hierarchical location of features within a network. By using a dynamic attention mechanism to calibrate cross-layer relationships, ACDA effectively mitigates negative transfer.

Future work may focus on reducing the computational overhead of all-to-all layer comparisons and extending the cross-layer alignment logic to dense prediction tasks like semantic segmentation and object detection, where spatial and semantic dislocation is particularly pronounced.

Summary updated

BibTeX
@article{ma2022attention,
  author = {Xu Ma and Junkun Yuan and Yen-Wei Chen and Ruofeng Tong and Lanfen Lin},
  title = {{Attention-based cross-layer domain alignment for unsupervised domain adaptation}},
  journal = {Neurocomputing},
  year = {2022},
  doi = {10.1016/j.neucom.2022.04.086},
}
Figure