Clinical Flair: A Pre-Trained Language Model for Spanish Clinical Natural Language Processing

Matías Rojas, Jocelyn Dunstan, Fabián VillenaNAACL 2022

TL;DR

This paper introduces "Clinical Flair," a novel pre-trained language model specifically designed for Spanish clinical natural language processing. By fine-tuning a general-domain Spanish language model on a corpus of clinical narratives, they demonstrate significant improvements in named entity recognition tasks, highlighting the crucial need for domain-specific models in under-resourced languages.

What Problem Does This Paper Address?

Natural Language Processing (NLP) has seen tremendous progress thanks to contextual word embeddings derived from large pre-trained language models. These models excel at capturing word meanings within their specific sentence context. However, while many such models exist for general-purpose English text, there's a notable gap for Spanish, especially in specialized domains like clinical text. Existing Spanish NLP models often rely on scientific articles, which differ significantly from the more informal and error-prone language found in real clinical narratives (like patient diagnoses). This lack of domain-specific models hinders the development of effective NLP tools for Spanish clinical data, which is abundant and crucial for healthcare applications.

Methods

The authors leverage the Flair framework, a character-level language model. Flair's strength lies in its ability to handle misspellings and out-of-vocabulary words by processing text at the character level, making it well-suited for noisy clinical data. Unlike models like BERT which rely on a fixed vocabulary and subword tokenization, Flair generates embeddings based on internal states of a bidirectional LSTM, capturing context from both directions.

The core of their contribution is fine-tuning pre-trained Spanish Flair models (es-forward and es-backward) on a specific clinical corpus. They used the Chilean Waiting List corpus, a collection of over 5 million free-text diagnostic suspicions from the Chilean public healthcare system. This fine-tuning process adapts the general language understanding of the pre-trained models to the nuances and terminology of Spanish clinical text.

For evaluating their model, they employed a standard LSTM-CRF (Long Short-Term Memory-Conditional Random Field) architecture, a common choice for sequence labeling tasks like Named Entity Recognition (NER). Their contextualized embeddings from Clinical Flair were integrated into the embedding layer of this LSTM-CRF model, replacing traditional word and character embeddings.

Datasets

Training Data: * Chilean Waiting List corpus: This is the primary dataset used for fine-tuning the Spanish Flair models. It comprises over 5 million free-text diagnostic suspicions from the Chilean public healthcare system, containing approximately 14 million sentences and 68 million tokens.

Evaluation Datasets: The authors evaluated their Clinical Flair model on four Spanish NER datasets from the clinical and biomedical domains:

  • CANTEMIST2: Focuses on oncologic clinical case reports, with mentions of tumor morphology.
  • PharmaCoNER: A biomedical corpus for recognizing pharmacological substances, compounds, and proteins.
  • Clinical Trials: Contains texts from abstracts of clinical trials and trial protocols, annotated with a subset of UMLS semantic groups.
  • NUBes: A biomedical corpus from anonymized health records, annotated for negation and uncertainty.

Key Insights and Ideas

  • Domain-Specific is Key: The most significant insight is the substantial performance boost achieved by using domain-specific embeddings (Clinical Flair) compared to general-domain embeddings (Spanish Flair). This underscores the importance of tailoring language models to the specific linguistic characteristics of a domain.
  • Character-Level Advantage: The choice of Flair as the base model proved beneficial due to its character-level processing, which naturally handles the misspellings and variations common in clinical text, a problem for vocabulary-based models like BERT.
  • Bridging the Gap for Spanish Clinical NLP: The paper effectively addresses a critical gap in Spanish NLP resources, providing a valuable tool for researchers and developers working with clinical data in this language.
  • Generalization Beyond Training Domain: While trained on clinical narratives, Clinical Flair also showed benefits on biomedical datasets, suggesting some degree of transferability, though not always outperforming models trained on more biomedical-specific corpora.

Key Results

The paper reports that Clinical Flair consistently outperformed both the general-domain Spanish Flair model and the SciELO Flair model (trained on scientific articles) across the four evaluated NER datasets.

  • CANTEMIST: Clinical Flair achieved an average F1-score difference of +0.028 compared to Spanish Flair and +0.005 compared to SciELO Flair.
  • Clinical Trials: The model showed an average F1-score difference of +0.023 compared to Spanish Flair and +0.012 compared to SciELO Flair.
  • PharmaCoNER: While not outperforming SciELO Flair, Clinical Flair achieved competitive results.
  • NUBes: The performance difference was smaller here (+0.007 compared to Spanish Flair), which the authors attribute to the dataset's focus on negation and uncertainty, making its target labels more general.

Overall, the results demonstrate a significant improvement, with Clinical Flair achieving superior performance by a "wide margin" in many cases.

Limitations and Future Work

Limitations: * While Clinical Flair showed benefits on biomedical datasets, it did not always outperform models specifically trained on biomedical corpora (e.g., SciELO Flair on PharmaCoNER). * The authors acknowledge that their model was trained on a specific type of clinical narrative (diagnoses) and might not perfectly capture all nuances of other clinical text types.

Future Work: * Extending the study to other NLP tasks beyond NER. * Exploring combinations of Clinical Flair embeddings with other embedding types (e.g., Word2vec, character-level embeddings). * Training a clinical version of BERT for Spanish to provide a wider range of contextual representation models. * Collecting a larger and more diverse clinical corpus to further improve performance, especially for models like BERT.

Summary updated

BibTeX
@inproceedings{rojas2022clinical,
  author = {Matías Rojas and Jocelyn Dunstan and Fabián Villena},
  title = {{Clinical Flair: A Pre-Trained Language Model for Spanish Clinical Natural Language Processing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure