Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation

Bo Zhang, Xiaoming Zhang, Yun Liu, Lei Cheng, Zhoujun Li

Abstract

This paper introduces Cross-domain Knowledge Distillation (CdKD), a framework for Unsupervised Domain Adaptation (UDA) that operates without access to original source data. By utilizing a novel metric called Joint Kernelized Stein Discrepancy (JKSD), the framework aligns a pre-trained source model with unlabeled target data. CdKD is uniquely architecture-agnostic, allowing for knowledge transfer between heterogeneous models (e.g., from BERT to CNN) by exploiting multi-view features, including class probabilities and per-sample gradients.

Introduction

Traditional Unsupervised Domain Adaptation (UDA) typically requires concurrent access to labeled source data and unlabeled target data. However, real-world scenarios often present two significant constraints: data privacy, where source data is restricted, and architectural rigidity, where the source and target models must share the same structure.

The authors propose a "source-free" UDA approach that addresses these hurdles. By treating the pre-trained source model as the sole representative of the source domain, they enable adaptation to new domains while remaining flexible about the target model's architecture.

Methodology

The core of the Cross-domain Knowledge Distillation (CdKD) framework is the integration of two primary objectives to bridge the domain gap:

  1. Knowledge Distillation (KD): The target model is trained to mimic the "soft" probability outputs of the source model when processing target data.
  2. Joint Kernelized Stein Discrepancy (JKSD): This is the mathematical foundation of the approach. Unlike metrics like Maximum Mean Discrepancy (MMD) that require samples from both domains, JKSD uses a Stein operator. This allows the framework to evaluate the discrepancy between the source model's learned distribution and the empirical target data using only the source model's score function (gradients of the log-density).
Overview of the CdKD framework, illustrating the transfer of multi-view features, the model pipeline, and the dual optimization objectives (JKSD and KnowDistill).

Furthermore, CdKD exploits gradient information as a rich, discriminative feature set. The authors argue that how a model's weights would need to change for a specific sample (the gradient) provides a finer-grained signal for adaptation than final class probabilities alone.

Experimental Setup

The framework was evaluated on two cross-domain text classification benchmarks:

  • Amazon-Feature: A dataset with 400-dimensional feature representations across four domains: Books (B), DVD (D), Electronics (E), and Kitchen (K).
  • Amazon-Text: A complex dataset of raw text used to test adaptation across different architectures, such as transferring knowledge from a BertGRU source to a TextCNN target.

Key Results and Discussion

CdKD demonstrated superior performance across various benchmarks. On the Amazon-Feature dataset, it achieved an average accuracy of 79.2%, outperforming both source-free competitors like SHOT (78.4%) and traditional methods that utilize source data.

Ablation study showing the performance of CdKD compared to its variants (CdKD-g, CdKD-a) and standard Knowledge Distillation (KD) across multiple domain transfer tasks.

As shown in the ablation studies, the full CdKD framework consistently outperforms standard KD and variants that only use a subset of features. Specifically, when adapting from a complex BertGRU to a TextCNN, CdKD improved accuracy by nearly 2% over standard KD, validating the importance of distribution matching via JKSD. The method also proved robust to variations in batch size and the initial accuracy of the source model.

Conclusion

CdKD provides a robust solution for source-free, architecture-agnostic domain adaptation. By leveraging the Stein operator and gradient-based features, it bypasses the need for source data while enabling the deployment of lightweight models on new domains. Future work may address the computational overhead of calculating joint distribution kernels and explore alignment in cases where latent space dimensions differ between models.

Summary updated

BibTeX
@inproceedings{zhang2021matching,
  author = {Bo Zhang and Xiaoming Zhang and Yun Liu and Lei Cheng and Zhoujun Li},
  title = {{Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.421},
}
Figure