Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios

Hazal Türkmen, Oguz Dikenelli, Cenk Eraslan, Mehmet Calli, Suha OzbekACL 2023

TL;DR

This paper explores how to best pre-train BERT models for Turkish clinical text, specifically for radiology reports, when data is scarce. They found that leveraging a large general-domain corpus, even with task-adaptive fine-tuning, yields superior results compared to solely relying on limited in-domain data, highlighting the importance of broad linguistic knowledge.

What Problem Does This Paper Address?

Natural Language Processing (NLP) has been revolutionized by large language models (LLMs) like BERT. However, applying these powerful models to specialized domains, especially in low-resource languages like Turkish, presents a significant challenge. Clinical domains, in particular, often have limited amounts of labeled data, making it difficult to train models that can accurately understand and process medical texts. This paper tackles the problem of adapting BERT for the Turkish clinical domain, specifically for analyzing radiology reports, by investigating different pre-training strategies when faced with limited data. They aim to determine the most effective ways to leverage both general and domain-specific data to achieve high performance.

Methods

The researchers explored two primary pre-training strategies:

  • Simultaneous Pre-training: This approach involves training a BERT model using a combination of a large general-domain corpus and a smaller, task-specific corpus. The idea is to balance broad linguistic understanding with domain-specific knowledge from the outset. They experimented with using an "amplified, domain-specific vocabulary" versus the standard BERTurk vocabulary.
  • Task-Adaptive Pre-training (Continual Pre-training): This method involves taking an already pre-trained general-domain model (like BERTurk) and further training it on a smaller, task-specific dataset. This allows the model to adapt its existing knowledge to the nuances of the target domain.

They developed four models: two using simultaneous pre-training (TurkRadBERT-sim v1 and v2) and two using task-adaptive pre-training (TurkRadBERT-task v1 and v2). The models differed in their initialization (general BERT vs. biomedical BERT) and vocabulary choices.

Datasets

The study utilized several corpora:

  • General Turkish Corpus: A large corpus (35 GB) used for training the BERTurk model.
  • Turkish Biomedical Corpus: A domain-specific corpus of medical articles.
  • Turkish Electronic Radiology Theses: A corpus of radiology theses.
  • Head CT Reports: A newly created, limited task-specific corpus of Turkish radiology reports (approximately 30 MB after preprocessing, with 2,000 reports used for annotation).

The Head CT Reports dataset was specifically curated for a multi-label classification task, aiming to identify 12 clinically significant observations and a "no findings" label. This dataset is noted as the first of its kind for multi-label classification in the Turkish clinical domain.

Key Insights and Ideas

  • General Domain Knowledge is Crucial: The study strongly suggests that leveraging a large, general-domain corpus during pre-training is vital, even when the ultimate goal is a specialized domain task. Models initialized with or trained on broad linguistic data (like BERTurk) consistently outperformed those relying more heavily on limited in-domain data alone.
  • Task-Adaptive Pre-training's Double-Edged Sword: While task-adaptive pre-training can help models capture domain-specific patterns, it risks overfitting when the task-specific corpus is very small. This can lead to a model that is too specialized and struggles to generalize.
  • Domain-Specific Vocabulary Matters (for Simultaneous Pre-training): For simultaneous pre-training, using an "amplified, domain-specific vocabulary" (TurkRadBERT-sim v1) showed better performance than using the general domain vocabulary (TurkRadBERT-sim v2). This indicates that tailoring the vocabulary can be beneficial when combining corpora.
  • Label Frequency Impacts Performance: The performance on specific labels can vary significantly, and labels with lower frequencies in the task-specific corpus might be more susceptible to noise and overfitting.

Key Results

  • Top Performers: BERTurk achieved the highest overall F1 score (0.9562), closely followed by TurkRadBERT-task v1 (0.9556). Both models leveraged extensive general-domain knowledge.
  • Simultaneous Pre-training Lag: The TurkRadBERT-sim models (v1 and v2) showed significantly lower performance (F1 scores of 0.8149 and 0.7879, respectively) compared to the task-adaptive models.
  • Category-Specific Differences: While BERTurk generally performed better, TurkRadBERT-task v1 outperformed BERTurk on specific labels like Hydrocephalus and Subarachnoid Bleeding, suggesting task-adaptive fine-tuning can offer boosts in certain areas.

Limitations and Future Work

Limitations:

  • Limited Data Size: The primary limitation is the scarcity of labeled clinical data, which inherently constrains the effectiveness of any model.
  • Overfitting Risk: Task-adaptive pre-training on small corpora carries a risk of overfitting, as acknowledged by the authors.
  • Imbalanced Datasets: The clinical datasets exhibit imbalanced class distributions, a common challenge in radiology text processing.

Future Work:

  • Expand Task-Specific Corpus: Increasing the size of the task-specific corpus could further improve domain-specific knowledge and performance on rare labels.
  • Explore Alternative Strategies: Investigating novel pre-training and fine-tuning techniques could enhance model performance.
  • Deeper Analysis of Label Differences: Further research into the factors contributing to performance variations across specific labels could yield valuable insights for developing more robust medical NLP models.

Summary updated

BibTeX
@inproceedings{turkmen2023harnessing,
  author = {Hazal Türkmen and Oguz Dikenelli and Cenk Eraslan and Mehmet Calli and Suha Ozbek},
  title = {{Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure