Pretrained Biomedical Language Models for Clinical NLP in Spanish

Casimiro Pio Carrino, Joan Llop, Marc Pàmies, Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Joaquín Silveira-Ocampo, Alfonso Valencia, Aitor Gonzalez-Agirre, Marta VillegasACL 2022

TL;DR

This paper introduces the first large-scale biomedical language models pre-trained from scratch specifically for Spanish clinical text. These models significantly outperform general-domain Spanish models on clinical Named Entity Recognition (NER) tasks, demonstrating the value of domain-specific pre-training for low-resource languages.

What Problem Does This Paper Address?

The field of clinical Natural Language Processing (NLP) has seen tremendous progress thanks to large pre-trained language models like BERT and its successors. However, most of this progress has been concentrated in English, leaving a significant gap for other languages, including Spanish. While general-domain Spanish models exist, they often fall short when applied to specialized domains like clinical text, which has its own unique vocabulary, syntax, and nuances. This paper addresses this gap by creating and evaluating dedicated biomedical and clinical language models for Spanish, aiming to improve NLP applications in this critical domain.

Methods

The authors pre-trained two Transformer-based language models from scratch using a RoBERTa architecture. They employed a Masked Language Modeling (MLM) objective with Subword Masking (SWM), similar to the original RoBERTa training. The models were trained on a massive corpus of Spanish biomedical and clinical text. For evaluation, they fine-tuned these models on three clinical Named Entity Recognition (NER) tasks. The fine-tuning process involved adding a standard linear layer for token classification and using the BIO tagging schema. They compared their models against general-domain Spanish models and existing domain-specific models that were created through continued pre-training on top of general models.

Datasets

The paper utilized two main corpora for training:

  • Biomedical Corpus: A large collection of 1.1 billion tokens from various Spanish sources, including:
  • A medical crawler of over 3,000 URLs from Spanish biomedical and health domains.
  • A collection of clinical cases.
  • Scientific publications from the Spanish SciELO server.
  • The BARR2 Background corpus for biomedical abbreviation recognition.
  • Wikipedia articles from the "Life Sciences" category.
  • Spanish patent data related to medical domains.
  • Documents from the European Medicines Agency (EMEA).
  • Spanish articles from MedlinePlus.
  • Open-access Spanish abstracts from PubMed.
  • EHR Corpus: A corpus of 95 million tokens from over 514,000 clinical documents, including discharge reports, clinical course notes, and X-ray reports.

For evaluation, they used three clinical NER datasets:

  • PharmaCoNER: A dataset for recognizing chemical and drug mentions from Spanish medical texts, derived from clinical case reports.
  • CANTEMIST: A shared task dataset for named entity recognition of tumor morphology in Spanish, based on oncological case reports.
  • ICTUSnet: A dataset of hospital discharge reports for stroke patients, containing annotations for various clinical variables.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: The results strongly suggest that pre-training language models from scratch on domain-specific data (biomedical and clinical Spanish) yields superior performance compared to using general-domain Spanish models.
  • "From Scratch" vs. "Continued Pre-training": The paper's findings indicate that when sufficient domain-specific data is available, pre-training from scratch can be more effective than continued pre-training on top of general-domain models for clinical tasks. This raises an interesting research question about the optimal pre-training strategy.
  • Robustness Matters: The authors' models exhibited smaller standard deviations in their performance across different configurations, suggesting they are more robust and reliable, which is a significant advantage when extensive hyper-parameter tuning is not feasible.
  • Addressing the Language Gap: This work makes a significant contribution to closing the NLP gap for Spanish in the biomedical and clinical domains, providing valuable resources for researchers and developers.

Key Results

The authors' pre-trained models, bsc-bio-es and bsc-bio-ehr-es, consistently outperformed general-domain Spanish models (like roberta-base-bne and mBERT) across all three clinical NER tasks. They also matched or surpassed existing domain-specific models that were created through continued pre-training, with the exception of the ICTUSnet dataset where XLM-R-Galén achieved the best development score. For instance, on the PharmaCoNER task, bsc-bio-es achieved an F1 score of 0.8907, while roberta-base-bne scored 0.8677. The models also demonstrated smaller standard deviations, indicating greater stability.

Limitations and Future Work

The authors acknowledge that while their models outperformed others on average, one specific model (XLM-R-Galén) achieved a better "best on development" score on the ICTUSnet dataset. They also note that their primary goal was to compare pre-training strategies rather than maximize NER task performance, and thus they did not employ more sophisticated classification layers.

Future work will focus on:

  • Further investigating the optimal pre-training approach (from scratch vs. continued pre-training).
  • Developing new models based on continued pre-training.
  • Expanding the evaluation to a wider range of clinical NLP tasks.
  • Making their pre-trained models and fine-tuning scripts publicly available to encourage further research and development in Spanish clinical NLP.

Summary updated

BibTeX
@inproceedings{carrino2022pretrained,
  author = {Casimiro Pio Carrino and Joan Llop and Marc Pàmies and Asier Gutiérrez-Fandiño and Jordi Armengol-Estapé and Joaquín Silveira-Ocampo and Alfonso Valencia and Aitor Gonzalez-Agirre and Marta Villegas},
  title = {{Pretrained Biomedical Language Models for Clinical NLP in Spanish}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure