Applications of BERT Models Towards Automation of Clinical Coding in Icelandic

Haraldur Orri Hauksson, Hafsteinn EinarssonNAACL 2024 · DOI

TL;DR

This paper explores automating clinical coding in Icelandic, a low-resource language, using BERT models. By fine-tuning pre-trained models on a large dataset of Icelandic electronic health records, they achieved performance comparable to state-of-the-art models trained on English datasets, demonstrating the potential for NLP in under-resourced healthcare systems.

What Problem Does This Paper Address?

Clinical coding, the process of assigning standardized codes (like ICD codes) to medical diagnoses and procedures, is crucial for healthcare operations, billing, and research. Traditionally, this is a manual, labor-intensive, and error-prone task. While Natural Language Processing (NLP), particularly Transformer-based models like BERT, has shown great promise in automating this for English, similar efforts for low-resource languages like Icelandic have been hindered by a lack of large, accessible datasets and pre-trained models. This paper aims to bridge this gap by investigating the effectiveness of BERT models for Icelandic clinical coding, leveraging a substantial, unique dataset.

Methods

The core of this research involves adapting and fine-tuning pre-trained language models for the Icelandic clinical coding task. Key methodological aspects include:

  • Continued Pre-training: They took an existing Icelandic BERT model (IceBERT) and further pre-trained it on the hospital's EHR data using a masked language modeling objective. This process, resulting in "MedIceBERT," aims to adapt the model to the specific domain of clinical text.
  • Model Adaptation: They explored different strategies for adapting the pre-trained models, including varying the size of the fine-tuning corpus and using a label attention mechanism (LAAT).
  • Label Attention (LAAT): This mechanism, introduced by Vu et al. (2021) and implemented as in Huang et al. (2022), is designed to help models focus on label-specific features, which is particularly useful for the large number of ICD codes.
  • Data Deduplication: To handle the large volume of EHR data and potential redundancy, they employed temporal deduplication and random subset deduplication techniques.

Datasets

  • Landspítali University Hospital (LUH) EHR Data: This is the primary dataset used, comprising over 10 million written records spanning 25 years (1997-2023). This dataset is unique as it covers various aspects of hospitalization, unlike some English datasets that focus solely on discharge summaries.
  • MIMIC-III/IV: These publicly available datasets are used for comparison with existing state-of-the-art results in English.

Key Insights and Ideas

  • Low-Resource Languages Can Compete: The study demonstrates that even languages with limited digital resources can achieve competitive clinical coding performance with the right approach, challenging the notion that only well-resourced languages can benefit from advanced NLP.
  • Domain-Specific Pre-training is Key: Further pre-training BERT models on domain-specific EHR data (MedIceBERT) significantly improved performance over a general-purpose Icelandic BERT model (IceBERT).
  • Label Attention is Crucial for Large Label Spaces: While LAAT showed marginal benefits on smaller label sets, it became essential for achieving good performance when dealing with a large number of ICD codes (8922 labels).
  • Temporal Deduplication is Effective: For large datasets, temporal deduplication proved to be an efficient strategy for reducing data size while preserving relevant information.

Key Results

  • Top 100 Codes: The best-performing model achieved a micro F1-score of 72.2 and a macro F1-score of 65.2. For AUC, the model achieved near-perfect scores (99.5 micro, 99.3 macro), outperforming English state-of-the-art models.
  • Top 8922 Codes: The model achieved a micro F1-score of 66.2 and a macro F1-score of 18.3. While the macro F1 is lower, the micro F1 is still competitive, especially considering the vast label space.
  • Effect of Pre-training: MedIceBERT consistently outperformed IceBERT by at least 3% in micro F1-score across various dataset sizes, highlighting the benefit of domain-specific pre-training.
  • Ablation Study: LAAT improved F1-scores and AUC, particularly for larger label spaces. Training on unlabeled data reduced F1-scores but increased AUC.

Limitations and Future Work

  • Dataset Quality: The inherent inaccuracies and limitations in EHR data representation and labeling could impact findings.
  • Computational Constraints: The study was limited to base models due to computational resources, suggesting larger or more complex models might yield different results. Experiments were also single runs, lacking error estimates.
  • Scalability: The scalability of the approach to longer texts and larger datasets remains untested.
  • Practical Utility Threshold: Determining the exact accuracy and reliability needed for effective clinical implementation is an open question.
  • Cross-Language Nuances: Direct comparisons with English models might not fully capture structural differences in datasets.
  • Potential for Bias: The models may inherit biases present in the dataset, which could lead to skewed outcomes.
  • Ethical and Operational Challenges: The paper acknowledges the broader ethical and operational implications of deploying AI in clinical settings, which require further investigation.

Future work suggested includes aggregating patient notes for a richer context, exploring advanced techniques for handling long sequences (e.g., Mamba, Hyena), and further investigating the ethical implications and human oversight needed for AI in clinical coding.

Summary updated

BibTeX
@inproceedings{hauksson2024applications,
  author = {Haraldur Orri Hauksson and Hafsteinn Einarsson},
  title = {{Applications of BERT Models Towards Automation of Clinical Coding in Icelandic}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.127},
}
Figure