Improving Self-training with Prototypical Learning for Source-Free Domain Adaptation on Clinical Text

Seiji Shimizu, Shuntaro Yada, Lisa Raithel, Eiji AramakiACL 2024

TL;DR

This paper introduces Source-Free Prototype-based Self-training (SFPS), a novel method to improve domain adaptation for clinical text without accessing the original training data. SFPS leverages prototypes generated from unlabeled target data to enhance self-training, achieving better performance than standard self-training methods.

What Problem Does This Paper Address?

In clinical Natural Language Processing (NLP), models trained on one dataset (source domain) often perform poorly when applied to a different dataset (target domain) due to domain shifts. This is a significant issue because sharing clinical data is restricted due to patient privacy concerns, making traditional domain adaptation methods that require access to source data unusable. Source-Free Domain Adaptation (SFDA) aims to address this by adapting models using only the trained source model and unlabeled target data. While self-training is a common SFDA technique, it can be unreliable due to noisy pseudo-labels. This paper tackles the challenge of improving SFDA by enhancing self-training with a prototype-based approach, specifically addressing how to generate reliable prototypes without source data access.

Methods

The core of this paper is the Source-Free Prototype-based Self-training (SFPS) framework. SFPS builds upon self-training by incorporating prototypical learning. The key innovation lies in generating prototypes without accessing the source data. The authors explore three source-free prototype generation methods:

  • Entropy-based (ENT): Selects samples with high entropy as prototypes.
  • Centroid-based (CEN): Uses the class centroid of target data representations as prototypes.
  • Class-weights-based (WGT): Leverages the weights of the source classifier as prototypes, augmented with similar target data representations.

These generated prototypes are then used in two ways:

  1. Prototype-based Pseudo-labeling: Assigns pseudo-labels to target data based on similarity to the generated prototypes, rather than solely relying on the source model's predictions.
  2. Contrastive Learning: Enhances target data representations by encouraging similarity with prototypes of the same class and dissimilarity with prototypes of different classes.

The overall SFPS objective function combines a pseudo-labeling loss, a contrastive learning loss, and a regularization loss that uses the original source model's predictions to prevent divergence.

Datasets

The authors evaluate their method on two clinical NLP tasks from SemEval-2021 Task 10:

  • Negation Detection: Using clinical notes from Mayo Clinic (source) and two target datasets: MIMIC and i2b2.
  • Time Expression Recognition: Using clinical notes from Mayo Clinic (source) and two target datasets: news articles (News) and food security warning systems data (Food).

The source models for these tasks were fine-tuned using English RoBERTa-base.

Key Insights and Ideas

  • Prototypical learning can significantly boost SFDA: The paper demonstrates that by intelligently generating and using prototypes, self-training can be made more robust and effective in a source-free setting.
  • Centroid-based prototype generation is a strong contender: Among the explored source-free methods, the centroid-based approach showed consistent improvements, suggesting it's a reliable way to capture class information from unlabeled target data.
  • SFPS effectively mitigates noisy pseudo-labels: The analysis shows that SFPS leads to higher accuracy and F1 scores for pseudo-labels compared to vanilla self-training, indicating it successfully filters out or corrects erroneous labels.
  • Contrastive learning and regularization are crucial: Ablation studies reveal that both the contrastive learning objective and the regularization term contribute positively to performance, highlighting the synergistic effect of the SFPS components.

Key Results

  • SFPS consistently outperformed vanilla self-training methods on most datasets, achieving the best performance in 3 out of 4 tested datasets.
  • The centroid-based (CEN) prototype generation method with a single iteration strategy was identified as the most reliable formulation, outperforming the un-updated source model on all datasets and the best vanilla self-training model on 3 out of 4.
  • For the MIMIC dataset in negation detection, SFPS (WGT with Single) achieved a 3.1-point higher F1 score than the best vanilla method.
  • In time expression recognition (News dataset), SFPS (ENT with RD+KM) improved F1 score by 0.8 points over the best vanilla method.
  • SFPS demonstrated higher pseudo-label accuracy and F1 scores compared to the source model and vanilla self-training across all datasets.

Limitations and Future Work

  • Limited Domain Generalizability: Experiments were conducted on clinical/English corpora, limiting conclusions for other domains.
  • Lack of Theoretical Analysis: The findings are based on empirical evidence, and a theoretical analysis is missing.
  • Performance Gap with Fully Fine-tuned Models: While SFPS improves upon vanilla self-training, a gap remains with fully fine-tuned models (Oracle), which is expected as SFPS does not use labeled target data.
  • Task and Dataset Scope: The experiments were limited to negation detection and time expression recognition. The authors suggest testing SFPS on other clinical domain tasks and datasets.
  • Future Work: The authors suggest exploring task-specific modules, such as class balancing, to further improve performance.

Summary updated

BibTeX
@inproceedings{shimizu2024improving,
  author = {Seiji Shimizu and Shuntaro Yada and Lisa Raithel and Eiji Aramaki},
  title = {{Improving Self-training with Prototypical Learning for Source-Free Domain Adaptation on Clinical Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure