Domain Adaptive Ensemble Learning

Kaiyang Zhou, Yongxin Yang, Yu Qiao, Tao Xiang

Abstract

The problem of generalizing deep neural networks from multiple source domains to a target one is studied under two settings: When unlabeled target data is available, it is a multi-source unsupervised domain adaptation (UDA) problem, otherwise a domain generalization (DG) problem. We propose a unified framework termed domain adaptive ensemble learning (DAEL) to address both problems. A DAEL model is composed of a CNN feature extractor shared across domains and multiple classifier heads each trained to specialize in a particular source domain. Each such classifier is an expert to its own domain and a non-expert to others. DAEL aims to learn these experts collaboratively so that when forming an ensemble, they can leverage complementary information from each other to be more effective for an unseen target domain. To this end, each source domain is used in turn as a pseudo-target-domain with its own expert providing supervisory signal to the ensemble of non-experts learned from the other sources. For unlabeled target data under the UDA setting where real expert does not exist, DAEL uses pseudo-label to supervise the ensemble learning. Extensive experiments on three multi-source UDA datasets and two DG datasets show that DAEL improves the state of the art on both problems, often by significant margins.

Introduction

Deep learning models are notoriously brittle when faced with domain shift—where the data they see at test time (the target domain) looks different from what they were trained on (the source domains). This challenge is typically addressed via Unsupervised Domain Adaptation (UDA), where unlabeled target data is available during training, or Domain Generalization (DG), where the target domain is entirely unknown.

Traditional approaches often focus on learning "domain-invariant" features by forcing the model to ignore domain-specific details. However, DAEL argues that domain-specific information, such as texture or lighting, is often useful for recognition. Discarding this information can weaken the model's overall predictive power.

The DAEL Framework

DAEL introduces a "multi-expert" architecture consisting of a single shared CNN backbone (the feature extractor) followed by multiple classifier heads. Each head is trained to become an "expert" for one specific source domain. This approach allows the model to preserve domain-specific patterns that can be combined later rather than erasing them in pursuit of invariance.

Overview of the DAEL framework showing (a) expert learning, (b) collaborative learning for labeled source data, and (c) collaborative learning for unlabeled target data.

Collaborative Ensemble Learning

The core innovation of DAEL is Collaborative Ensemble Learning, which uses a "pseudo-target" strategy to prepare the model for unseen domains.

Learning from Labeled Source Data

During training, the model treats each source domain in turn as a "pseudo-target." The expert specifically trained for that domain provides supervision (in the form of soft probability labels) to an ensemble of the other experts, referred to as the "non-expert group." This process forces the non-experts to learn how to handle data from a domain they were not specifically optimized for, effectively simulating the challenge of domain shift.

Learning from Unlabeled Target Data (UDA)

In the UDA setting, where no ground-truth expert exists for the target domain, DAEL identifies the most confident prediction from the existing source experts to generate a pseudo-label. This pseudo-label then serves as the supervisory signal for the entire ensemble.

Consistency Regularization

To enhance robustness, DAEL employs a "weak-strong" augmentation strategy. The supervising expert (or most confident expert) processes a weakly augmented version of an image to provide a stable signal, while the ensemble being trained processes a strongly augmented version. The ensemble is then tasked with matching the expert's prediction through a consistency loss.

Detailed diagrams of the consistency loss mechanisms for both labeled source domains (left) and unlabeled target domains (right).

Experimental Evaluation

The authors evaluated DAEL on several standard benchmarks and introduced miniDomainNet, a smaller version of the massive DomainNet dataset designed for faster prototyping.

  • UDA Performance: On the Digit-5 dataset, DAEL reached 96.47% accuracy, nearly matching the "Oracle" performance (97.00%) where the model is trained on labeled target data. On the full DomainNet, DAEL outperformed the previous state-of-the-art (CMSS) by 2.2% on average.
  • Domain Generalization: In DG tasks where the target is never seen during training, DAEL achieved superior results on PACS (83.4%) and Office-Home (66.1%).

Conclusion

DAEL demonstrates that collaboration between domain-specific experts is a powerful alternative to seeking domain-invariant features. By rotating which domain acts as the "target" during training, the model practices for real-world domain shifts. While the method's success is somewhat dependent on the choice of data augmentation strategies, its ability to achieve state-of-the-art results across both UDA and DG settings marks a significant advancement in robust machine learning.

Summary updated

Abstract

The problem of generalizing deep neural networks from multiple source domains to a target one is studied under two settings: When unlabeled target data is available, it is a multi-source unsupervised domain adaptation (UDA) problem, otherwise a domain generalization (DG) problem. We propose a unified framework termed domain adaptive ensemble learning (DAEL) to address both problems. A DAEL model is composed of a CNN feature extractor shared across domains and multiple classifier heads each trained to specialize in a particular source domain. Each such classifier is an expert to its own domain and a non-expert to others. DAEL aims to learn these experts collaboratively so that when forming an ensemble, they can leverage complementary information from each other to be more effective for an unseen target domain. To this end, each source domain is used in turn as a pseudo-target-domain with its own expert providing supervisory signal to the ensemble of non-experts learned from the other sources. For unlabeled target data under the UDA setting where real expert does not exist, DAEL uses pseudo-label to supervise the ensemble learning. Extensive experiments on three multi-source UDA datasets and two DG datasets show that DAEL improves the state of the art on both problems, often by significant margins. The code is released at \url{https://github.com/KaiyangZhou/Dassl.pytorch}.
BibTeX
@article{zhou2021domain,
  author = {Kaiyang Zhou and Yongxin Yang and Yu Qiao and Tao Xiang},
  title = {{Domain Adaptive Ensemble Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=7F0wwMQDQ0u},
}
Figure