Feature Alignment and Restoration for Domain Generalization and Adaptation

Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo ChenarXiv 2020 · arXiv

Abstract

Domain transfer techniques often rely on feature alignment to bridge the gap between source and target domains. However, standard alignment methods frequently discard task-relevant discriminative details in the pursuit of domain invariance. This paper presents the Feature Alignment and Restoration (FAR) framework, which adaptively aligns features while explicitly restoring task-critical information from the discarded residuals. By employing a dual ranking entropy loss, FAR ensures that restored features enhance classification confidence. Experimental results across multiple benchmarks demonstrate that FAR achieves state-of-the-art performance in both unsupervised domain adaptation and domain generalization.

Introduction

In machine learning, "domain shift" occurs when a model trained on one type of data (e.g., professional photos) fails on another (e.g., hand-drawn sketches). To mitigate this, researchers typically employ Feature Alignment (FA), which forces the model to learn domain-invariant representations. However, alignment is often "task-ignorant." In the process of making features from different domains match, the model may discard discriminative details that are domain-specific yet crucial for the task.

Motivation: Standard alignment can lose discriminative details like texture, making it difficult to distinguish between similar shapes like a horse and a sheep across different domains.

As illustrated in Figure 1, if a model relies solely on "shape" to align photos and sketches, it might lose the "texture" information necessary to distinguish a horse from a sheep in the photo domain. The FAR framework is designed to overcome this trade-off by recovering these lost signals.

Methodology: The FAR Framework

The proposed FAR framework addresses the limitations of standard alignment through a two-phase process involving adaptive alignment and information restoration.

1. Feature Alignment (FA)

Rather than aligning all features indiscriminately, FAR utilizes spatial and channel attention to adaptively select a subspace of features most suitable for alignment. It then matches the moments (mean and variance) of these distributions across domains. This selective approach preserves more raw information for the subsequent restoration phase compared to global alignment strategies.

2. Feature Restoration (FR)

This core component calculates a "residual" — the difference between the original features and the aligned ones. An attention module distills task-relevant information from this residual, which is then added back to the aligned features to compensate for lost discriminative power. This ensures the final representation is both domain-invariant and task-optimized.

Dual Ranking Entropy Loss

To ensure the restoration step captures useful information rather than domain-specific noise, the authors introduce a Dual Ranking Entropy Loss. This loss encourages the enhanced feature (aligned + task-relevant residual) to yield lower entropy, indicating higher prediction confidence. Conversely, it forces a contaminated feature (aligned + task-irrelevant residual) to yield higher entropy, indicating lower confidence. This mechanism supervises the disentanglement of relevant features without requiring explicit labels for what constitutes "relevance."

Experimental Evaluation

The FAR framework was evaluated on four major benchmarks: Digit-Five, mini-DomainNet, PACS, and Office-Home.

Key Results

FAR consistently outperformed existing state-of-the-art (SOTA) methods across both adaptation and generalization tasks:

  • Unsupervised Domain Adaptation (UDA): On the Digit-Five benchmark, FAR achieved an average accuracy of 90.39%, outperforming the previous SOTA (M3SDA-β) by 2.74%. On mini-DomainNet, it reached 57.03%, a 1.51% improvement over the next best method.
  • Domain Generalization (DG): FAR achieved top results on PACS (81.7%) and Office-Home (66.0%), demonstrating robust generalization to completely unseen domains.

Qualitative analysis via t-SNE visualizations indicates that FAR produces cleaner, more separable class clusters compared to standard alignment methods, which often cause class overlap in the shared feature space as they force alignment at the expense of class boundaries.

Conclusion

The FAR framework demonstrates that information discarded during alignment is not merely noise but often contains task-critical details. By treating the alignment residual as a source for restoration and using entropy ranking for disentanglement, FAR effectively bridges the domain gap without sacrificing discriminative accuracy. Future work will explore extending this framework to dense prediction tasks such as image segmentation and object detection.

Summary updated

Abstract

For domain generalization (DG) and unsupervised domain adaptation (UDA), cross domain feature alignment has been widely explored to pull the feature distributions of different domains in order to learn domain-invariant representations. However, the feature alignment is in general task-ignorant and could result in degradation of the discrimination power of the feature representation and thus hinders the high performance. In this paper, we propose a unified framework termed Feature Alignment and Restoration (FAR) to simultaneously ensure high generalization and discrimination power of the networks for effective DG and UDA. Specifically, we perform feature alignment (FA) across domains by aligning the moments of the distributions of attentively selected features to reduce their discrepancy. To ensure high discrimination, we propose a Feature Restoration (FR) operation to distill task-relevant features from the residual information and use them to compensate for the aligned features. For better disentanglement, we enforce a dual ranking entropy loss constraint in the FR step to encourage the separation of task-relevant and task-irrelevant features. Extensive experiments on multiple classification benchmarks demonstrate the high performance and strong generalization of our FAR framework for both domain generalization and unsupervised domain adaptation.
BibTeX
@misc{jin2020feature,
  author = {Xin Jin and Cuiling Lan and Wenjun Zeng and Zhibo Chen},
  title = {{Feature Alignment and Restoration for Domain Generalization and Adaptation}},
  year = {2020},
  eprint = {2006.12009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.12009},
}
Figure