When Multilingual Models Compete with Monolingual Domain-Specific Models in Clinical Question Answering

Vojtech Lanz, Pavel PecinaNAACL 2025

TL;DR

This paper investigates whether general-purpose multilingual language models can effectively perform clinical question answering (QA) in languages lacking specialized medical models. By augmenting an English clinical QA dataset with translations into six European languages, the authors demonstrate that multilingual models can achieve competitive performance, even outperforming domain-specific English models in some cases, challenging the assumption that specialized pretraining is always superior.

What Problem Does This Paper Address?

Medical professionals spend significant time sifting through lengthy clinical documents to find specific patient information. While Question Answering (QA) systems can streamline this process, most existing research and models are heavily focused on English. This creates a significant gap for healthcare systems in non-English speaking regions, where clinical texts are in local languages, and English-centric models are ineffective. The paper aims to bridge this gap by exploring the potential of multilingual language models to support clinical QA in a wider range of languages, particularly those without dedicated medical NLP resources.

Methods

The core of the paper involves a comprehensive pipeline for creating a multilingual clinical QA dataset and evaluating multilingual models.

  • Machine Translation (MT) Pipeline: The authors translate an English clinical QA dataset (emrQA) into six European languages (Bulgarian, Czech, Greek, Spanish, Polish, Romanian). This process involves translating both clinical reports and questions. A key challenge addressed is aligning the answer evidence (spans of text) from the original English to the translated text.
  • Prompted Medical Phrase (PMP) Approach: To improve translation quality, especially for clinical jargon and abbreviations, they introduce a PMP approach. This involves prepending a phrase like "Based on medical reports" to segments before translation, providing explicit context to the MT model.
  • Answer Evidence Alignment: After translation, they employ word alignment models (FastAlign and Awesome) to project answer evidence from the source to the target language. A novel unsupervised forward-backward substring alignment evaluation method is introduced to assess the quality of these alignments without requiring labeled data.
  • Model Fine-tuning and Evaluation: Several multilingual models (mBERT, distilmBERT, XLM-R, XLM-R Large) are fine-tuned on the augmented multilingual dataset. They are evaluated in both monolingual (per language) and multilingual settings. The performance is compared against monolingual, domain-specific English models.

Datasets

  • emrQA Dataset: The primary dataset used is emrQA, a clinical QA dataset derived from the n2c2 challenge dataset. The authors utilize two subsets: "Medication" and "Relations."
  • Translated emrQA Datasets: The paper generates new datasets by translating the emrQA subsets into Bulgarian (BG), Czech (CS), Greek (EL), Spanish (ES), Polish (PL), and Romanian (RO).
  • NLLB Parallel Corpus: Used for training the FastAlign alignment model.

Key Insights and Ideas

  • Multilingual Models Compete: General-domain multilingual models can perform surprisingly well on clinical QA tasks, even without specific medical pretraining in those target languages.
  • Clinical Pretraining Not Always Superior: Contrary to expectations, domain-specific English models are not always superior to general-domain multilingual models on clinical QA tasks. In some instances, multilingual models even outperform their specialized English counterparts.
  • Data Augmentation is Key: Translating existing English clinical QA data into other languages significantly boosts the performance of multilingual models, demonstrating the value of multilingual data augmentation in the clinical domain.
  • Language-Specific Challenges: The translation and QA process presents varying levels of difficulty across different languages, influenced by factors like linguistic similarity to English and the quality of MT models for those language pairs.
  • Alignment Quality Matters: The quality of aligning answer evidence substrings in translated text is crucial and directly impacts QA performance. The proposed unsupervised evaluation method helps in filtering low-quality alignments.

Key Results

  • Competitive Performance: Multilingual models, particularly mBERT and XLM-R, achieved strong performance on the emrQA dataset across multiple languages, often closing the gap with English performance.
  • Outperforming Domain-Specific Models: For the "Medication" subset, multilingual models consistently outperformed monolingual, domain-specific English models (BERTbase, ClinicalBERT, BioBERT) in terms of EM and F1 scores.
  • Impact of PMP: The Prompted Medical Phrase (PMP) approach showed improvements in translation quality and subsequent alignment scores compared to standard MT, especially for challenging clinical texts.
  • Filtering Benefits: Removing the lowest-quality answer evidence alignments (around 15%) from the training data led to improved QA scores, suggesting that imperfect translations can hinder performance.
  • Relations vs. Medication: The "Relations" subset generally showed higher performance and less improvement from multilingual augmentation compared to the "Medication" subset, possibly because the "Relations" task is closer to an "oracle" (easier to answer) or has less room for improvement.

Limitations and Future Work

  • Dataset Quality: The study acknowledges limitations due to the quality of the emrQA dataset itself, which is synthetically generated and may contain inaccuracies.
  • Single Clinical Task: The conclusions are based on a specific clinical QA task (finding answer evidence), and their generalizability to a broader range of clinical NLP tasks or languages needs further investigation.
  • Future Work: The authors suggest exploring a broader range of clinical tasks and languages, and further investigating the impact of different MT and alignment strategies. They also highlight the potential for multilingual models to support low-resource languages in clinical settings.

Summary updated

BibTeX
@inproceedings{lanz2025when,
  author = {Vojtech Lanz and Pavel Pecina},
  title = {{When Multilingual Models Compete with Monolingual Domain-Specific Models in Clinical Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure