Abstract
This paper introduces Cross-domain Knowledge Distillation (CdKD), a framework for Unsupervised Domain Adaptation (UDA) that operates without access to original source data. By utilizing a novel metric called Joint Kernelized Stein Discrepancy (JKSD), the framework aligns a pre-trained source model with unlabeled target data. CdKD is uniquely architecture-agnostic, allowing for knowledge transfer between heterogeneous models (e.g., from BERT to CNN) by exploiting multi-view features, including class probabilities and per-sample gradients.
Introduction
Traditional Unsupervised Domain Adaptation (UDA) typically requires concurrent access to labeled source data and unlabeled target data. However, real-world scenarios often present two significant constraints: data privacy, where source data is restricted, and architectural rigidity, where the source and target models must share the same structure.
The authors propose a "source-free" UDA approach that addresses these hurdles. By treating the pre-trained source model as the sole representative of the source domain, they enable adaptation to new domains while remaining flexible about the target model's architecture.
Methodology
The core of the Cross-domain Knowledge Distillation (CdKD) framework is the integration of two primary objectives to bridge the domain gap:
- Knowledge Distillation (KD): The target model is trained to mimic the "soft" probability outputs of the source model when processing target data.
- Joint Kernelized Stein Discrepancy (JKSD): This is the mathematical foundation of the approach. Unlike metrics like Maximum Mean Discrepancy (MMD) that require samples from both domains, JKSD uses a Stein operator. This allows the framework to evaluate the discrepancy between the source model's learned distribution and the empirical target data using only the source model's score function (gradients of the log-density).
Furthermore, CdKD exploits gradient information as a rich, discriminative feature set. The authors argue that how a model's weights would need to change for a specific sample (the gradient) provides a finer-grained signal for adaptation than final class probabilities alone.
Experimental Setup
The framework was evaluated on two cross-domain text classification benchmarks:
- Amazon-Feature: A dataset with 400-dimensional feature representations across four domains: Books (B), DVD (D), Electronics (E), and Kitchen (K).
- Amazon-Text: A complex dataset of raw text used to test adaptation across different architectures, such as transferring knowledge from a BertGRU source to a TextCNN target.
Key Results and Discussion
CdKD demonstrated superior performance across various benchmarks. On the Amazon-Feature dataset, it achieved an average accuracy of 79.2%, outperforming both source-free competitors like SHOT (78.4%) and traditional methods that utilize source data.
As shown in the ablation studies, the full CdKD framework consistently outperforms standard KD and variants that only use a subset of features. Specifically, when adapting from a complex BertGRU to a TextCNN, CdKD improved accuracy by nearly 2% over standard KD, validating the importance of distribution matching via JKSD. The method also proved robust to variations in batch size and the initial accuracy of the source model.
Conclusion
CdKD provides a robust solution for source-free, architecture-agnostic domain adaptation. By leveraging the Stein operator and gradient-based features, it bypasses the need for source data while enabling the deployment of lightweight models on new domains. Future work may address the computational overhead of calculating joint distribution kernels and explore alignment in cases where latent space dimensions differ between models.
Summary updated