Unsupervised Domain Adaptation of Black-Box Source Models

Haojian Zhang, Yabin Zhang, Kui Jia, Lei Zhang

Abstract

Unsupervised domain adaptation (UDA) aims to learn models for a target domain of unlabeled data by transferring knowledge from a labeled source domain. Traditional UDA assumes labeled source data are available. Due to privacy concerns, source-free UDA (White-Box UDA) has emerged, assuming only a trained source model is available. However, source models may be proprietary or kept as black boxes for security. This work introduces Black-Box Unsupervised Domain Adaptation (B$^2$UDA), where only the API of the source model is accessible. We propose Iterative Learning with Noisy Labels (IterLNL), which treats black-box predictions as noisy labels and iteratively refines a target model using noise-rate estimation and category-wise sampling. Experiments demonstrate that IterLNL performs comparably to traditional UDA methods despite having significantly less information.

Introduction: The Black-Box Constraint

In modern machine learning, "Model-as-a-Service" is a common paradigm. Users often interact with high-performing models via APIs without access to internal weights or the original training data. This creates a barrier for standard Unsupervised Domain Adaptation (UDA) techniques that rely on feature alignment or gradient-based fine-tuning.

Comparison of UDA settings: Traditional UDA vs. White-Box UDA (WBUDA) vs. Black-Box UDA (B$^2$UDA).

As shown in Figure 1, B$^2$UDA is the most restrictive yet realistic setting. The challenge lies in adapting to a target domain when the source model is a "black box" that only provides output labels or probabilities.

Methodology: Iterative Learning with Noisy Labels (IterLNL)

The core insight of this work is to reframe domain adaptation as a Learning with Noisy Labels (LNL) problem. Since the source model is not perfectly accurate on the target domain, its predictions serve as "noisy labels." IterLNL addresses this through three key components:

  1. Noise Rate Estimation: To train effectively, the algorithm must estimate the proportion of incorrect labels. Because deep models are often overconfident, a rescale curve is used to map raw confidence scores to more accurate noise estimates.
  2. Category-wise Sampling: Domain shift affects classes differently. To prevent the model from ignoring difficult classes, IterLNL employs a category-wise sampling strategy using a probability queue buffer, ensuring balanced learning across all labels.
  3. Iterative Strategy: The process is iterative. A target model trained on filtered noisy labels often surpasses the source API's accuracy. This improved model then generates better labels for the next round of training, effectively "bootstrapping" its way to high performance.
Visualization of label noise patterns (Pair, Symmetry, VisDA-2017) and the rescale curve used for noise estimation.

Experimental Evaluation

The authors evaluated IterLNL on benchmark datasets including Office-31, VisDA-2017, and Digits.

  • VisDA-2017: IterLNL achieved 80.1% accuracy, a significant improvement over the source model's initial 49.6%. It outperformed previous black-box methods like sMDA by nearly 27%.
  • Office-31: The method reached an average accuracy of 87.9%, rivaling top-tier white-box methods (e.g., SHOT) that require full access to model weights.

Conclusion

IterLNL demonstrates that high-quality domain adaptation is possible even when the source model is entirely opaque. By treating domain shift as structured label noise and using iterative refinement, the method bridges the gap between black-box accessibility and white-box performance. This has significant implications for privacy-preserving AI and the practical deployment of commercial models in new environments.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) aims to learn models for a target domain of unlabeled data by transferring knowledge from a labeled source domain. In the traditional UDA setting, labeled source data are assumed to be available for adaptation. Due to increasing concerns for data privacy, source-free UDA is highly appreciated as a new UDA setting, where only a trained source model is assumed to be available, while labeled source data remain private. However, trained source models may also be unavailable in practice since source models may have commercial values and exposing source models brings risks to the source domain, e.g., problems of model misuse and white-box attacks. In this work, we study a subtly different setting, named Black-Box Unsupervised Domain Adaptation (B$^2$UDA), where only the application programming interface of source model is accessible to the target domain; in other words, the source model itself is kept as a black-box one. To tackle B$^2$UDA, we propose a simple yet effective method, termed Iterative Learning with Noisy Labels (IterLNL). With black-box models as tools of noisy labeling, IterLNL conducts noisy labeling and learning with noisy labels (LNL), iteratively. To facilitate the implementation of LNL in B$^2$UDA, we estimate the noise rate from model predictions of unlabeled target data and propose category-wise sampling to tackle the unbalanced label noise among categories. Experiments on benchmark datasets show the efficacy of IterLNL. Given neither source data nor source models, IterLNL performs comparably with traditional UDA methods that make full use of labeled source data.
BibTeX
@inproceedings{zhang2021unsupervised,
  author = {Haojian Zhang and Yabin Zhang and Kui Jia and Lei Zhang},
  title = {{Unsupervised Domain Adaptation of Black-Box Source Models}},
  booktitle = {BMVC},
  year = {2021},
}
Figure