Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis

Ali Abedi, Q. M. Jonathan Wu, Ning Zhang, Farhad PourpanaharXiv 2026 · DOI · arXiv

TL;DR

The CAT-CKD framework introduces a parameter-efficient way to adapt skin cancer diagnostic models to new clinical settings without ever needing to see the original training data. By combining lightweight convolutional adapters with a "consensus" distillation strategy, the researchers achieved state-of-the-art performance (86.1% AUROC) while training 95% fewer parameters than traditional methods.

What Problem Does This Paper Address?

Deep learning models for skin cancer diagnosis often struggle with domain shift. A model trained on high-quality dermoscopic images from one hospital might fail when deployed at a clinic using different cameras, lighting conditions, or patient demographics.

While "Domain Adaptation" techniques exist to fix this, most require access to the original source data. In the medical world, privacy regulations (like HIPAA or GDPR) often make sharing raw patient images between institutions impossible. This leads us to Multi-Source-Free Domain Adaptation (MSFDA)—the challenge of adapting to a new "target" clinic using only pre-trained models from several "source" clinics, without the source images themselves.

Existing MSFDA methods have three major flaws:

  1. Catastrophic Forgetting: Fine-tuning an entire large model on a small new dataset often causes it to "forget" its general knowledge.
  2. Feature Mismatch: They often try to align data at the end of the pipeline rather than building a shared, domain-invariant foundation.
  3. Noisy Labels: They rely on "hard" pseudo-labels (e.g., "this is definitely melanoma"), which propagate errors if the teacher models are even slightly wrong.

Methods

The authors propose CAT-CKD, a two-stage framework:

1. Contrastive Adapter Training (CAT)

Instead of fine-tuning the entire Vision Transformer (ViT) backbone, they keep it frozen. They insert ConvPass adapters—tiny convolutional bypass modules—into the ViT layers.

  • The Innovation: They train these adapters using Supervised Contrastive Learning (SCL) on pooled source data. This forces the model to group "melanoma" features together and "nevus" features together, regardless of which clinic the image came from. This creates a "domain-invariant" feature space before any clinic-specific training happens.

2. Consensus Knowledge Distillation (CKD)

Once the backbone is ready, they train individual "teacher" models for each source clinic. To adapt to the new, unlabeled target clinic:

  • The Innovation: They don't use hard labels. Instead, they take the raw output scores (logits) from all teacher models and average them to create a consensus signal.
  • A "student" model is then trained to match this consensus using KL Divergence. This "soft" approach is more stable because it accounts for uncertainty; if the teachers disagree, the student isn't forced to make a confident (and potentially wrong) guess.

Datasets

The study utilized five prominent public skin lesion datasets to simulate a multi-center clinical environment:

  • ISIC 2018 & ISIC 2020: Large-scale dermoscopic image sets.
  • PAD-UFES-20: Images collected via smartphones (representing a significant domain shift).
  • Derm7pt-Derm & Derm7pt-Clinic: A dataset split by imaging modality (dermoscopic vs. standard clinical photography).

The researchers used a "leave-one-domain-out" protocol, where four datasets served as labeled sources and the fifth acted as the unlabeled target.

Key Insights and Ideas

  • Adapters are "Anti-Forgetting" Shields: By freezing the backbone and only training 0.3M parameters in the CAT phase, the model retains the powerful general features learned from ImageNet while specializing in skin textures.
  • The Power of the Average: The paper includes a mathematical proof (Lemma 1) showing that averaging logits from multiple teachers reduces the variance of the error. In plain English: the "wisdom of the crowd" filters out the individual biases of specific clinics.
  • Visualizing Success: Attention maps showed that without CAT, the model looked at random background noise. With CAT, the model's "eyes" shifted significantly toward the actual skin lesions.

Key Results

  • Top-Tier Accuracy: CAT-CKD achieved an average 86.1% AUROC across all tasks, outperforming established methods like DANN, ADDA, and even recent MSFDA specialists like EAAF.
  • Extreme Efficiency: The model requires only 4.3 million trainable parameters. For context, competing methods often require fine-tuning over 90 million parameters.
  • Robustness to "Hard" Shifts: On the PAD-UFES-20 dataset (smartphone images), which showed the highest domain gap via Fréchet Distance analysis, CAT-CKD maintained a significant lead over other source-free methods.

Limitations and Future Work

While CAT-CKD performs well, the authors note that extreme domain gaps (like moving from professional dermoscopy to low-res smartphone photos) still result in a performance drop compared to easier transfers.

Future directions mentioned include:

  • Efficiency: Making the source-domain training phase even faster.
  • Security: Testing how the framework handles adversarial attacks.
  • Federated Learning: Extending the approach to settings where models are trained across decentralized servers to further enhance privacy.

Summary updated

Abstract

Skin cancer diagnosis, particularly differentiating melanoma from benign nevi, is challenging due to the visual similarity between lesions. Although convolutional neural networks (CNNs) and vision transformers (ViTs) have shown promising performance, their effectiveness often deteriorates on data from heterogeneous clinical sources. Conventional domain adaptation methods address domain shift but require access to source data during adaptation, which is often infeasible due to privacy regulations. Multi-source-free unsupervised domain adaptation (MSFDA) addresses this limitation by leveraging multiple labeled source domains to generalize to an unlabeled target domain without requiring access to source data. However, existing MSFDA methods rely on full backbone fine-tuning, leading to catastrophic forgetting and overfitting on small clinical datasets, and address domain shift at the aggregation stage without establishing a shared domain-invariant feature space. Furthermore, their reliance on hard pseudo-labels or confidence-weighted aggregation introduces noisy supervision signals under domain shift. To address these limitations, we propose CAT-CKD, consisting of two components: (1) contrastive adapter training (CAT), which trains lightweight ConvPass adapters within a frozen ViT backbone using supervised contrastive learning (SCL) to establish a shared domain-invariant feature space before source-specific model training, and (2) consensus knowledge distillation (CKD), which aggregates logits from multiple source models into a consensus supervisory signal and adapts a student model on unlabeled target data using KL divergence. Experiments on five publicly available skin lesion datasets show that CAT-CKD achieves an average AUROC of 86.1%, outperforming existing MSFDA methods while requiring only 4.3M trainable parameters. The code is available at https://github.com/A-Abedi/CAT_CKD.
BibTeX
@misc{abedi2026contrastive,
  author = {Ali Abedi and Q. M. Jonathan Wu and Ning Zhang and Farhad Pourpanah},
  title = {{Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis}},
  year = {2026},
  eprint = {2611.03528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03528},
}
Figure