TL;DR
This paper investigates the optimal language model (LM) strategy for specialized healthcare applications, specifically electronic pathology reports. It finds that finetuning smaller, domain-specific LMs (BiLMs) often outperforms zero-shot large LMs (LLMs), especially for complex tasks. Further domain-specific pretraining also provides significant benefits for challenging scenarios.
What Problem Does This Paper Address?
The rapid evolution of language models presents a dilemma for practitioners: which model architecture and adaptation strategy should be chosen for a given task? This is particularly critical in specialized domains like healthcare, where accuracy and efficiency are paramount. The authors aim to provide guidance by systematically evaluating the trade-offs between:
- Finetuning vs. Zero-Shot: Should models be adapted to the specific task with labeled data, or can they perform well "out-of-the-box"?
- Domain-Adjacent vs. Generic Models: Is it better to use models pre-trained on general text, or those pre-trained on related (e.g., clinical) text?
- Further Domain-Specific Pretraining: Does additional pretraining on data from the target institution further enhance performance?
- Small LMs (BiLMs) vs. Large LMs (LLMs): Are traditional BERT-style models still relevant and competitive against the newer, massive LLMs?
Methods
The study employs a comparative evaluation framework using three distinct scenarios derived from electronic pathology reports. They assess various language models, including:
- Bidirectional Language Models (BiLMs):
- General-purpose: RoBERTa (125M parameters)
- Domain-adjacent (clinical): PathologyBERT (108M parameters), Gatortron (345M parameters)
- Domain-specific further pretraining: RoBERTa and Gatortron were further pre-trained on 1 million pathology reports from the British Columbia Cancer Registry (BCCR), denoted as BCCROBERTa and BCCRTron.
- Large Language Model (LLM): Mistral-NeMo-Instruct (12B parameters), evaluated in a zero-shot capacity.
Models were evaluated using both zero-shot (no task-specific training) and finetuned (weights updated on task-specific data) approaches, where applicable. Performance was measured using macro-averaged F1 scores.
Datasets
The primary dataset used is electronic pathology reports from the British Columbia Cancer Registry (BCCR). This dataset was utilized to create three distinct evaluation scenarios:
- Scenario a (Easy): Reportability classification, a binary classification task with a large training dataset (40,000 reports) and a test set of 20,400 reports.
- Scenario b (Medium Hard): Tumor group classification, a multi-class task with a limited training dataset (16,000 reports) and a test set of 2,058 reports.
- Scenario c (Hard): Histology classification for leukemia, a multi-class task with a small training dataset (1,000 reports) and a test set of 447 reports.
The paper also mentions using a corpus of 1 million pathology reports from BCCR for further domain-specific pretraining of the BiLMs.
Key Insights and Ideas
- Finetuning is King for BiLMs: For BiLMs, finetuning is not just beneficial, it's critical. Zero-shot performance for BiLMs is consistently very low, while finetuning dramatically boosts their accuracy across all scenarios.
- LLMs Shine in Zero-Shot, But Not Always: LLMs demonstrate superior zero-shot capabilities compared to BiLMs. However, they were consistently outperformed by finetuned BiLMs on these specific healthcare classification tasks.
- Domain Matters: Domain-adjacent pre-trained models (like PathologyBERT and Gatortron) generally outperform generic models (like RoBERTa) after finetuning, especially on more challenging tasks.
- Further Pretraining Pays Off (Especially for Hard Tasks): While further pretraining on domain-specific data offers modest gains on easier tasks, it provides significant improvements for complex, data-scarce scenarios like the leukemia classification task.
- BiLMs Aren't Dead: Despite the rise of LLMs, BiLMs remain highly relevant and effective for specialized, well-defined tasks like classification, particularly when finetuning data is available. They can offer a better performance-resource trade-off.
Key Results
- Finetuned BiLMs vs. Zero-Shot LLM:
- In Scenario a (easy, large data), finetuned BiLMs achieved F1 scores of 0.95-0.97, while the zero-shot LLM reached 0.76.
- In Scenario b (medium, limited data), finetuned BiLMs achieved 0.78-0.85, compared to the LLM's 0.54.
- In Scenario c (hard, small data), finetuned BiLMs achieved 0.60-0.78, while the LLM reached 0.65.
- Domain-Adjacent vs. Generic BiLMs: Finetuned domain-adjacent models (PathologyBERT, Gatortron) generally outperformed the generic RoBERTa, especially in harder tasks.
- Impact of Further Pretraining: For Scenario c, BCCROBERTa (0.71) and BCCRTron (0.89) significantly outperformed their non-further-pretrained counterparts (RoBERTa: 0.61, Gatortron: 0.78).
Limitations and Future Work
The authors acknowledge that their evaluation is based on a "small set of models" and that the results are specific to the tested scenarios and datasets. They suggest that future work could explore a wider range of LLMs, different adaptation strategies for LLMs (beyond zero-shot), and more diverse healthcare NLP tasks. The paper's core contribution is providing empirical evidence to guide practitioners in making informed decisions about LM selection in specialized domains, highlighting the continued value of finetuned BiLMs.
Summary updated