Multi-Source Domain Adaptation via supervised contrastive learning and confident consistency regularization

Marin Scalbert, Florent Couzinié-Devy, Maria Vakalopoulou

TL;DR

Deep learning models often struggle when deployed in new environments due to "domain shift." This paper introduces Contrastive Multi-Source Domain Adaptation (CMSDA), a novel framework that combines interpolated supervised contrastive learning to align multiple labeled source domains, and consistency regularization to adapt to an unlabeled target domain. By effectively aligning class distributions across domains, CMSDA achieves state-of-the-art performance on several standard benchmarks.

What Problem Does This Paper Address?

When you train a model on one dataset (the source) and test it on another (the target), performance usually drops because the data distributions differ—a phenomenon known as Domain Shift. Unsupervised Domain Adaptation (UDA) tries to fix this by leveraging labeled source data and unlabeled target data. In the most practical scenarios, you might have multiple different labeled source domains (Multi-Source UDA).

Most existing Multi-Source UDA methods try to align the marginal distributions of the source and target domains, or they match explicit statistics like class centroids. However, these approaches often fail to guarantee that the class conditional distributions are aligned (i.e., making sure a "dog" in domain A maps to the same feature space as a "dog" in domain B). Centroid matching also assumes a class can be represented by a single point, which fails if the class has a multimodal distribution. This paper aims to robustly align these class conditional distributions across multiple sources while simultaneously adapting to the target domain.

Methods

The authors propose a unified architecture with a shared feature extractor, a projection head, and a classification head. The training process is split into two main branches:

  1. Source Domain Alignment (MixUp + Interpolated Contrastive Learning): To learn domain-invariant features, the model uses Supervised Contrastive Learning (SCL) on the source data. SCL pulls representations of the same class together and pushes different classes apart. To make the model even more robust, the authors apply MixUp (interpolating random pairs of source images and their labels). Because MixUp creates "soft" labels, standard SCL (which requires hard labels) breaks. The authors solve this by using an Interpolated Supervised Contrastive Loss (ISCL) that can handle these mixed examples.

  2. Target Domain Adaptation (Consistency Regularization): For the unlabeled target data, the model relies on semi-supervised learning techniques—specifically, a FixMatch-style approach. The model generates a "weakly augmented" and a "strongly augmented" version of a target image. If the model's prediction on the weak version is highly confident, it uses that prediction as a "hard pseudo-label" to train the network on the strongly augmented version.

Datasets

The framework is evaluated on three standard Multi-Source UDA image classification benchmarks:

  • DomainNet: The largest UDA benchmark, containing ~0.6 million images across 6 distinct domains (clipart, infograph, painting, quickdraw, real, sketch) and 345 classes.
  • MiniDomainNet: A smaller, less noisy subset of DomainNet featuring 4 domains and 126 classes (~140K images).
  • Office-Home: A widely used dataset containing ~15,500 images across 4 domains (Art, Clipart, Product, Real-World) and 65 classes.

Key Insights and Ideas

  • MixUp as Style Mixing: Applying MixUp across different source domains acts as a highly effective data augmentation strategy. It essentially "mixes" the styles of different domains, forcing the model to learn domain-invariant features rather than memorizing domain-specific textures.
  • Contrastive Learning Solves Conditional Alignment: By using supervised contrastive learning on the source domains, the model naturally aligns class conditional distributions. A "car" from the Clipart domain is forced into the same latent space as a "car" from the Real-World domain, creating highly transferable features.
  • Additive Gains: The ablation studies reveal that the performance gains from the source-side contrastive loss and the target-side consistency regularization are largely independent and additive.
  • Preventing Negative Transfer: On highly challenging target domains (like DomainNet's "Quickdraw"), models often perform worse than if they had just trained on the source data alone (negative transfer). CMSDA successfully avoids this, maintaining robust performance even under severe domain shift.

Key Results

  • State-of-the-Art Performance: CMSDA achieves the best average accuracy on all three datasets.
  • DomainNet: 50.42% average accuracy (+1.72% over the previous SOTA, DAEL).
  • MiniDomainNet: 61.90% average accuracy.
  • Office-Home: 76.60% average accuracy.
  • Ablation Impact: Adding the Interpolated Supervised Contrastive Loss (ISCL) provided significant boosts on the hardest domains, yielding up to a +2.48% accuracy jump on the DomainNet Quickdraw target and +2.75% on the Office-Home Real-World target compared to using cross-entropy alone.

Limitations and Future Work

  • Hyperparameter Sensitivity: The model relies heavily on several hyperparameters, notably the contrastive temperature ($T$), the MixUp interpolation strength ($\alpha$), and the pseudo-label confidence threshold ($\tau$). The authors note that setting the temperature too high or the confidence threshold too low can introduce noisy pseudo-labels and degrade performance.
  • Future Directions: Currently, the contrastive loss is only applied to the labeled source data. The authors suggest that future work could apply supervised contrastive learning to both the source data and the pseudo-labeled target data to align all conditional distributions simultaneously. They also plan to explore domain generalization augmentations like MixStyle or Fourier-based augmentations.

Summary updated

Abstract

Multi-Source Unsupervised Domain Adaptation (multi-source UDA) aims to learn a model from several labeled source domains while performing well on a different target domain where only unlabeled data are available at training time. To align source and target features distributions, several recent works use source and target explicit statistics matching such as features moments or class centroids. Yet, these approaches do not guarantee class conditional distributions alignment across domains. In this work, we propose a new framework called Contrastive Multi-Source Domain Adaptation (CMSDA) for multi-source UDA that addresses this limitation. Discriminative features are learned from interpolated source examples via cross entropy minimization and from target examples via consistency regularization and hard pseudo-labeling. Simultaneously, interpolated source examples are leveraged to align source class conditional distributions through an interpolated version of the supervised contrastive loss. This alignment leads to more general and transferable features which further improves the generalization on the target domain. Extensive experiments have been carried out on three standard multi-source UDA datasets where our method reports state-of-the-art results.
BibTeX
@inproceedings{scalbert2021multi,
  author = {Marin Scalbert and Florent Couzinié-Devy and Maria Vakalopoulou},
  title = {{Multi-Source Domain Adaptation via supervised contrastive learning and confident consistency regularization}},
  booktitle = {BMVC},
  year = {2021},
}

Loading related content...

Figure