TL;DR
This paper introduces Source-Free Prototype-based Self-training (SFPS), a novel method to improve domain adaptation for clinical text without accessing the original training data. SFPS leverages prototypes generated from unlabeled target data to enhance self-training, achieving better performance than standard self-training methods.
What Problem Does This Paper Address?
In clinical Natural Language Processing (NLP), models trained on one dataset (source domain) often perform poorly when applied to a different dataset (target domain) due to domain shifts. This is a significant issue because sharing clinical data is restricted due to patient privacy concerns, making traditional domain adaptation methods that require access to source data unusable. Source-Free Domain Adaptation (SFDA) aims to address this by adapting models using only the trained source model and unlabeled target data. While self-training is a common SFDA technique, it can be unreliable due to noisy pseudo-labels. This paper tackles the challenge of improving SFDA by enhancing self-training with a prototype-based approach, specifically addressing how to generate reliable prototypes without source data access.
Methods
The core of this paper is the Source-Free Prototype-based Self-training (SFPS) framework. SFPS builds upon self-training by incorporating prototypical learning. The key innovation lies in generating prototypes without accessing the source data. The authors explore three source-free prototype generation methods:
- Entropy-based (ENT): Selects samples with high entropy as prototypes.
- Centroid-based (CEN): Uses the class centroid of target data representations as prototypes.
- Class-weights-based (WGT): Leverages the weights of the source classifier as prototypes, augmented with similar target data representations.
These generated prototypes are then used in two ways:
- Prototype-based Pseudo-labeling: Assigns pseudo-labels to target data based on similarity to the generated prototypes, rather than solely relying on the source model's predictions.
- Contrastive Learning: Enhances target data representations by encouraging similarity with prototypes of the same class and dissimilarity with prototypes of different classes.
The overall SFPS objective function combines a pseudo-labeling loss, a contrastive learning loss, and a regularization loss that uses the original source model's predictions to prevent divergence.
Datasets
The authors evaluate their method on two clinical NLP tasks from SemEval-2021 Task 10:
- Negation Detection: Using clinical notes from Mayo Clinic (source) and two target datasets: MIMIC and i2b2.
- Time Expression Recognition: Using clinical notes from Mayo Clinic (source) and two target datasets: news articles (News) and food security warning systems data (Food).
The source models for these tasks were fine-tuned using English RoBERTa-base.
Key Insights and Ideas
- Prototypical learning can significantly boost SFDA: The paper demonstrates that by intelligently generating and using prototypes, self-training can be made more robust and effective in a source-free setting.
- Centroid-based prototype generation is a strong contender: Among the explored source-free methods, the centroid-based approach showed consistent improvements, suggesting it's a reliable way to capture class information from unlabeled target data.
- SFPS effectively mitigates noisy pseudo-labels: The analysis shows that SFPS leads to higher accuracy and F1 scores for pseudo-labels compared to vanilla self-training, indicating it successfully filters out or corrects erroneous labels.
- Contrastive learning and regularization are crucial: Ablation studies reveal that both the contrastive learning objective and the regularization term contribute positively to performance, highlighting the synergistic effect of the SFPS components.
Key Results
- SFPS consistently outperformed vanilla self-training methods on most datasets, achieving the best performance in 3 out of 4 tested datasets.
- The centroid-based (CEN) prototype generation method with a single iteration strategy was identified as the most reliable formulation, outperforming the un-updated source model on all datasets and the best vanilla self-training model on 3 out of 4.
- For the MIMIC dataset in negation detection, SFPS (WGT with Single) achieved a 3.1-point higher F1 score than the best vanilla method.
- In time expression recognition (News dataset), SFPS (ENT with RD+KM) improved F1 score by 0.8 points over the best vanilla method.
- SFPS demonstrated higher pseudo-label accuracy and F1 scores compared to the source model and vanilla self-training across all datasets.
Limitations and Future Work
- Limited Domain Generalizability: Experiments were conducted on clinical/English corpora, limiting conclusions for other domains.
- Lack of Theoretical Analysis: The findings are based on empirical evidence, and a theoretical analysis is missing.
- Performance Gap with Fully Fine-tuned Models: While SFPS improves upon vanilla self-training, a gap remains with fully fine-tuned models (Oracle), which is expected as SFPS does not use labeled target data.
- Task and Dataset Scope: The experiments were limited to negation detection and time expression recognition. The authors suggest testing SFPS on other clinical domain tasks and datasets.
- Future Work: The authors suggest exploring task-specific modules, such as class balancing, to further improve performance.
Summary updated