Transfer Learning for Low-Resource Clinical Named Entity Recognition

Nevasini Sasikumar, Krishna Sri Ipsit MantriACL 2023

TL;DR

This paper introduces a novel transfer learning approach to adapt high-resource English clinical Named Entity Recognition (NER) models to low-resource languages and domains. By translating target domain data into English, fine-tuning the English model, and then transferring it back, they achieve significant performance gains with minimal annotated data, making clinical NLP more accessible globally.

What Problem Does This Paper Address?

Clinical text, like physician notes and discharge summaries, is rich with information crucial for healthcare. However, unlocking this data through Natural Language Processing (NLP) is challenging, especially in clinical settings where annotated data is scarce, particularly for languages other than English and for specialized domains like patient dialogues. Traditional deep learning models for NLP are data-hungry, making it difficult to achieve high performance in these low-resource scenarios. This paper tackles the problem of enabling high-performance clinical NER in these challenging, low-resource environments by proposing a method that leverages existing high-resource models.

Methods

The core of the proposed method is a transfer learning strategy that involves several key steps:

  1. High-Resource Model Training: A BERT-based NER model is first trained on a large English clinical dataset (nearly 1 million EHR notes). This establishes a strong foundation in clinical NER.
  2. Data Translation: Datasets from low-resource target languages or domains (Spanish, French, clinical dialogues) are translated into English using an automated machine translation system (Google Cloud Translate). This creates "weakly annotated" data, which may contain noise but provides a substantial amount for fine-tuning.
  3. Fine-tuning on Translated Data: The pre-trained English clinical NER model is then fine-tuned on these machine-translated English datasets. This step adapts the model's knowledge to the nuances of the target domain, even though the training data is in English. Techniques like dropout and weight freezing are explored to prevent overfitting.
  4. Transfer to Target Domain: Finally, the fine-tuned model is used for inference on data in the original target language or domain. Domain adaptation techniques are employed to bridge the gap between the translated training data and the original inference data.

This approach contrasts with methods that require direct, extensive annotation in the target language/domain.

Datasets

The paper utilizes the following datasets:

  • English Clinical Dataset: Approximately 950,000 anonymized EHR notes from Anthropic, PBC, used for training the initial high-resource English NER model.
  • Target Datasets:
    • Spanish clinical notes (10,000 notes)
    • French clinical notes (20,000 notes)
    • Clinical dialogue text (50,000 notes)

These target datasets were translated into English for the fine-tuning step. Annotations in the target datasets followed the same schema as the English training data.

Key Insights and Ideas

  • Translation as a Bridge: The paper demonstrates that machine translation, despite its potential for noise, can serve as an effective bridge to adapt models to new domains and languages when direct annotation is limited.
  • Leveraging High-Resource Models: A strong, pre-trained model in a data-rich domain (English clinical NER) can be a powerful starting point for low-resource scenarios, significantly outperforming models trained solely on limited target data.
  • Domain Adaptation is Crucial: The success of transferring the fine-tuned model back to the original target domain highlights the importance of domain adaptation techniques to bridge the gap between the translated training data and the actual inference data.
  • Efficiency Gains: The method offers substantial time and cost savings compared to traditional expert annotation, making clinical NLP more scalable and accessible.

Key Results

The transferred models consistently outperformed models trained solely on the target datasets:

  • Spanish: Achieved a 10.5% gain in F1 score (84.2% F1 for target-only vs. 94.7% F1 for transferred).
  • French: Achieved a 5.5% gain in F1 score (87.3% F1 for target-only vs. 92.8% F1 for transferred).
  • Clinical Dialogue: Achieved a 4.3% gain in F1 score (82.1% F1 for target-only vs. 86.4% F1 for transferred).

Qualitative analysis revealed that the transferred models exhibited stronger generalization, better handling of linguistic complexity (e.g., long noun phrases, ambiguous abbreviations), and reduced false positives by focusing on relevant clinical context. The method was also found to be more time and cost-efficient, requiring significantly less human annotation effort.

Limitations and Future Work

The authors acknowledge the following limitations and suggest future directions:

  • Limitations:
    • The reliance on machine translation introduces potential noise into the training data.
  • The current work focuses on NER; extending to other clinical NLP tasks is a natural next step.
  • Future Work:
    • Zero-shot Transfer: Explore methods that require no target annotations at all.
  • Automated Dataset Selection: Develop reinforcement learning for optimal dataset selection for translation and fine-tuning.
  • Advanced Domain Adaptation: Investigate more sophisticated techniques like parameter shuffling and adversarial training.
  • Broader Task Coverage: Expand to other clinical tasks such as relation extraction, topic classification, and summarization.
  • Multi-task Learning: Explore transfer learning across multiple clinical domains and languages simultaneously.
  • Open-Source Resources: Release models and datasets to create a benchmark for low-resource clinical NLP.

The overarching goal is to make high-performance clinical NLP more accessible, particularly in underserved settings, by advancing transfer learning and domain adaptation techniques.

Summary updated

BibTeX
@inproceedings{sasikumar2023transfer,
  author = {Nevasini Sasikumar and Krishna Sri Ipsit Mantri},
  title = {{Transfer Learning for Low-Resource Clinical Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure