TL;DR
This paper investigates how to perform Named Entity Recognition (NER) on clinical text in languages other than English, where annotated data is scarce. They compare cross-lingual transfer (CLT) with translation-based methods, finding that while CLT with large models often performs best, translation-based methods can be competitive and offer more flexibility, especially with smaller models.
What Problem Does This Paper Address?
Extracting information from clinical texts, like identifying drug names or dosages (Named Entity Recognition or NER), is crucial for structuring medical reports. However, most existing NER models and datasets are in English. Developing these models for other languages is challenging due to the high cost and effort required to create annotated datasets. This paper tackles this problem by exploring methods to perform clinical NER in French and German without relying on any French or German annotated data, leveraging existing English resources.
Methods
The authors compare three main approaches for cross-lingual NER:
- Cross-lingual Transfer (CLT): This involves fine-tuning a multilingual language model (like XLM-R) on an English clinical NER dataset and then directly applying it to French and German texts.
- Translate-Train: Here, the English training data is translated into the target language (French or German), and then an NER model is trained on this translated data. This involves using machine translation models and alignment algorithms to project annotations.
- Translate-Test: In this method, the target language test data is translated into English, an English NER model is applied, and the predicted labels are then projected back to the original language.
They also explore the impact of using domain-specific language models and investigate techniques like "realignment" to improve representations of multilingual models.
Datasets
- n2c2 (Track 2, Adverse Drug Events and Medication Extraction): An English clinical dataset used for training and fine-tuning models.
- MedNERF: A newly released French clinical NER dataset, created from drug prescriptions and annotated with the same guidelines as n2c2. This serves as a test set for French experiments.
- GERNERMED test set: A German clinical NER dataset used for evaluation, consisting of 30 sentences from physicians, annotated similarly to n2c2.
- BioWMT19 challenge corpus, UFAL dataset, Medline: Parallel medical texts used for fine-tuning translation and alignment models.
Key Insights and Ideas
- Translation vs. Transfer Trade-offs: While large multilingual models (like XLM-R Large) often achieve the best performance with CLT, translation-based methods can be competitive and offer more flexibility, especially when using smaller models.
- Model Size Matters: Smaller models (like distilmBERT) benefit more from translation-based methods, particularly translate-test, suggesting they might perform better as monolingual models rather than in a cross-lingual transfer setting.
- Alignment is Crucial: The choice of alignment model in translation-based methods is critical and can significantly impact downstream NER performance, sometimes more so than the translation model itself.
- Domain-Specific Models Aren't Always Better: While domain-specific models can perform well, they don't consistently outperform large general-purpose multilingual models like XLM-R, especially if the domain-specific models were trained on significantly less data.
- Fine-tuning Translation Models: Fine-tuning translation models on in-domain medical data can improve their intrinsic translation quality, but this doesn't always translate to better NER performance. Choosing a fundamentally different translation model (e.g., FAIR vs. Opus) can have a larger impact.
Key Results
- CLT with Large Models: XLM-R Large consistently achieved the highest F1 scores for both French and German using CLT, outperforming translation-based methods in many cases.
- Translate-Train Competitiveness: Translate-train, especially with pre-selected translation and alignment models, achieved comparable results to CLT, sometimes even surpassing it, particularly when using specific alignment strategies.
- Translate-Test Performance: Translate-test generally performed worse than translate-train, especially for larger models, as errors in translation and alignment directly impact the test set. However, it showed promise with smaller models like distilmBERT.
- Realignment Benefits: Realignment of multilingual model representations showed significant improvements in some cases, allowing CLT to outperform translate-train baselines.
- Computational Efficiency: CLT and translate-train have similar inference times, while translate-test is slower due to on-the-fly translation. Translate-train has a longer training time due to the initial translation step.
Limitations and Future Work
- Limited Languages: The study focused only on French and German. Future work could explore other low-resource languages.
- Encoder-Only Architectures: The research was limited to encoder-only models. Investigating generative models with zero-shot learning approaches would be a valuable next step.
- Dataset Differences: The MedNERF dataset (prescriptions) differs from n2c2 and GERNERMED (clinical reports). While the authors found this domain shift didn't significantly impact results, it's a factor to consider.
- Release of Datasets: The authors advocate for the release of more non-English clinical datasets to facilitate future research.
- Prompt Engineering: For future zero-shot approaches, careful prompt selection strategies would be necessary.
Summary updated