Get the Best out of 1B LLMs: Insights from Information Extraction on Clinical Documents

Saeed Farzi, Soumitra Ghosh, Alberto Lavelli, Bernardo MagniniACL 2024

TL;DR

This paper investigates how smaller, 1-billion parameter language models (LLMs) perform on clinical information extraction tasks (Named Entity Recognition and Relation Extraction) when fine-tuned, continuously pre-trained on medical data, or instruction-tuned. The findings highlight the effectiveness of domain-specific pre-training (MedMT5) and suggest that while instruction tuning (FLAN-T5) is promising, it may not always surpass domain-specific approaches, especially in low-resource, multilingual settings.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen a revolution with the advent of large language models (LLMs) like BERT and GPT. These models excel at various tasks, but their application in specialized domains like clinical text presents unique challenges. Clinical documents are often scarce, making it difficult to train models from scratch. Furthermore, privacy concerns in healthcare settings limit the use of cloud-based solutions, necessitating models that can run on standard hardware like single GPUs. While large, general-purpose LLMs are powerful, the paper questions their optimal use in resource-constrained, domain-specific scenarios, particularly when compared to smaller, specialized models. The research aims to provide insights into the best strategies for adapting LLMs for clinical information extraction, considering factors like fine-tuning, domain-specific pre-training, and instruction tuning, especially in a multilingual context.

Methods

The study compares four versions of 1-billion parameter generative models:

  • T5: A base text-to-text transformer model.
  • mT5: A multilingual version of T5, pre-trained on a diverse corpus across 101 languages.
  • MedMT5: mT5 further pre-trained on a medical domain corpus in four languages.
  • FLAN-T5: A T5 model instruction-tuned on a wide range of general NLP tasks.

These models were evaluated on two core information extraction tasks:

  1. Clinical Entity Detection (NER): Identifying clinical entities like patient information, admission dates, and medical history.
  2. Test-Result Relation Extraction (RE): Identifying relationships between laboratory tests and their measurements.

The researchers employed a text-to-text generation approach for both tasks. They also explored data augmentation techniques, specifically cross-lingual translation, to assess its impact on performance.

Datasets

The study utilized the following datasets:

  • European Clinical Case Corpus (E3C): A multilingual dataset of clinical cases covering Spanish, Basque, English, French, and Italian. For NER, English and Italian subsets were used.
  • NCBI Disease Corpus: A dataset of disease mentions in PubMed abstracts, used for NER.
  • CLinkaRT and TESTLINK datasets: Italian and Spanish datasets, respectively, focused on relation extraction between clinical laboratory tests and their results.

Key Insights and Ideas

  • Domain-Specific Pre-training Shines: MedMT5, which was pre-trained on a medical corpus, consistently outperformed the other T5 variants on clinical tasks, especially in the Italian language. This strongly suggests the value of domain-specific knowledge for clinical NLP.
  • Instruction Tuning is Promising, But Not Always Superior: FLAN-T5, the instruction-tuned model, showed strong performance, particularly on English NER, but it didn't consistently outperform domain-specific pre-training (MedMT5) across all tasks and languages.
  • Multilinguality Matters: mT5's multilingual capabilities were beneficial, but its performance on English-only tasks was sometimes surpassed by the English-specialized T5. This highlights the trade-offs between broad language coverage and domain-specific optimization.
  • Data Augmentation's Mixed Impact: While data augmentation (especially translation-based) showed improvements in specific metrics (e.g., precision for RE), its overall impact on F1 scores was often marginal and dataset-dependent. It can enhance certain aspects but doesn't always lead to a significant leap in overall performance.
  • Smaller Models are Viable: The study demonstrates that 1-billion parameter models, when properly adapted, can achieve competitive results in clinical information extraction, making them suitable for resource-constrained environments and privacy-sensitive applications.

Key Results

  • NER Performance: MedMT5 achieved the highest F1 scores on E3C-Italian, significantly improving precision and recall. FLAN-T5 achieved the highest F1 score on E3C-English. For the NCBI dataset, T5 showed strong precision and recall, with MedMT5 and FLAN-T5 performing similarly.
  • RE Performance: MedMT5 generally outperformed other T5 family members in relation extraction across both Italian and Spanish, with mT5 showing a slight precision advantage in Italian. Data augmentation led to a substantial boost in precision for relation extraction in both languages.
  • Error Analysis: Models struggled with interpreting abbreviations and sometimes incorrectly grouped phrases as single entities, indicating a need for better contextual understanding and abbreviation recognition.

Limitations and Future Work

  • Dataset Complexity: The E3C corpus is noted as being more complex than the NCBI dataset, contributing to performance variations.
  • Entity Types: The NER experiments focused on a single entity type, and future work will explore performance across various entity types.
  • Language Coverage: Differences in language coverage between instruction-tuned and domain-specific models might influence results, particularly in multilingual settings.
  • Future Work: The authors plan to investigate parameter-optimized strategies (PEFT, LORA, QLORA, LLAMA-Adapter) for training larger models efficiently, explore sliding window approaches for handling long input sentences in RE, and conduct a more comprehensive analysis of different T5 models across various entity types. They also aim to investigate the impact of window size in future research.

Summary updated

BibTeX
@inproceedings{farzi2024get,
  author = {Saeed Farzi and Soumitra Ghosh and Alberto Lavelli and Bernardo Magnini},
  title = {{Get the Best out of 1B LLMs: Insights from Information Extraction on Clinical Documents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure