Alleviating Imbalanced Pseudo-label Distribution: Self-Supervised Multi-Source Domain Adaptation with Label-specific Confidence

Shuai Lü, Meng Kang, Ximing Li

Abstract

The existing self-supervised Multi-Source Domain Adaptation (MSDA) methods often suffer an imbalanced characteristic among the distribution of pseudo-labels. Such imbalanced characteristic results in many labels with too many or too few pseudo-labeled samples on the target domain, referred to as easy-to-learn label and hard-to-learn label, respectively. Both of these labels hurt the generalization performance on the target domain. To alleviate this problem, in this paper we propose a novel multi-source domain adaptation method, namely Self-Supervised multi-Source Domain Adaptation with Label-specific Confidence (S3DA-LC). Specifically, we estimate the label-specific confidences, i.e., the learning difficulties of labels, and adopt them to generate the pseudo-labels for target samples, enabling to simultaneously constrain and enrich the pseudo supervised signals for easy-to-learn and hard-to-learn labels. We evaluate S3DA-LC on several benchmark datasets, indicating its superior performance compared with the existing MSDA baselines.

Introduction

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Multi-Source Domain Adaptation (MSDA) extends this by leveraging multiple source domains, a scenario more representative of real-world applications but significantly more complex to coordinate. A common strategy in MSDA is self-supervised pseudo-labeling, where the model generates predictions for target data to serve as training signals.

However, existing methods frequently encounter the problem of imbalanced pseudo-label distribution. As shown in Figure 1, certain classes are inherently easier for the model to recognize across domains, leading to a "rich get richer" phenomenon. These "easy-to-learn" classes dominate the pseudo-label pool with potentially noisy samples, while "hard-to-learn" classes are neglected, severely hindering the model's ability to generalize across the entire target label space.

Figure 1: The imbalanced distribution of pseudo-labels in the DomainNet dataset, highlighting the disparity between easy and hard-to-learn classes.

Proposed Method: $S^3DA-LC$

To address this imbalance, we introduce Self-Supervised multi-Source Domain Adaptation with Label-specific Confidence ($S^3DA-LC$). The method operates in two distinct stages:

1. Warm-up Stage

The model first trains a shared feature extractor alongside domain-specific classifiers for each source. It calculates transferability weights for each source using Shannon entropy. If a source classifier demonstrates high certainty regarding the target data, it is assigned a higher weight in the final ensemble prediction.

2. Adaptation Stage with Label-specific Confidence

The core innovation lies in replacing fixed confidence thresholds with adaptive, label-specific thresholds ($\delta_c$).

  • Estimating Difficulty: The model quantifies learning difficulty by comparing the average source label distribution (the "prior") to the predicted distribution on the target domain.
  • Adaptive Thresholding: For under-represented "hard" classes, the threshold is lowered to encourage the inclusion of more samples. Conversely, for over-represented "easy" classes, the threshold is raised to filter out noise.
  • Outlier Handling: We employ Tukey’s Fences to ensure that extreme distribution shifts do not result in inaccurate thresholds.
Figure 2: Visualization of the relationship between class difficulty (V/G) and the adaptive thresholds (delta) calculated by S3DA-LC.

As illustrated in Figure 2, the model dynamically adjusts the confidence threshold $\delta$ based on the estimated difficulty $V/G$ for each class, effectively balancing the supervised signals.

Experimental Results

We evaluated $S^3DA-LC$ on three major benchmarks: Office-31, Office-Home, and the large-scale DomainNet. Our method consistently outperformed state-of-the-art (SOTA) baselines:

  • Office-31: Achieved 92.4% average accuracy (vs. 91.6% for CAiDA).
  • Office-Home: Reached 80.7% average accuracy, a significant improvement over the previous SOTA of 77.4%.
  • DomainNet: On this high-diversity dataset, $S^3DA-LC$ achieved 54.8% accuracy.

Ablation studies confirmed that the label-specific confidence mechanism is the primary driver of performance. For instance, removing $\delta_c$ on the Office-Home "Clipart" task caused accuracy to drop from 70.0% to 57.5%.

Conclusion and Limitations

$S^3DA-LC$ effectively mitigates pseudo-label imbalance by using source distributions as a guide for target learning. While highly effective, the method assumes a degree of similarity between source and target label distributions and requires careful tuning of the scaling parameter $\tau$. Future work will explore extending this adaptive thresholding logic to source-free domain adaptation scenarios.

Summary updated

Abstract

The existing self-supervised Multi-Source Domain Adaptation (MSDA) methods often suffer an imbalanced characteristic among the distribution of pseudo-labels. Such imbalanced characteristic results in many labels with too many or too few pseudo-labeled samples on the target domain, referred to as easy-to-learn label and hard-to-learn label, respectively. Both of these labels hurt the generalization performance on the target domain. To alleviate this problem, in this paper we propose a novel multi-source domain adaptation method, namely Self-Supervised multi-Source Domain Adaptation with Label-specific Confidence (S3DA-LC). Specifically, we estimate the label-specific confidences, i.e., the learning difficulties of labels, and adopt them to generate the pseudo-labels for target samples, enabling to simultaneously constrain and enrich the pseudo supervised signals for easy-to-learn and hard-to-learn labels. We evaluate S3DA-LC on several benchmark datasets, indicating its superior performance compared with the existing MSDA baselines.
BibTeX
@inproceedings{lu2024alleviating,
  author = {Shuai Lü and Meng Kang and Ximing Li},
  title = {{Alleviating Imbalanced Pseudo-label Distribution: Self-Supervised Multi-Source Domain Adaptation with Label-specific Confidence}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=CIlauWi8JG},
}
Figure