RadLing: Towards Efficient Radiology Report Understanding

Rikhiya Ghosh, Oladimeji Farri, Sanjeev Kumar Karn, Manuela Danu, Ramya Vunikili, Larisa MicuACL 2023

TL;DR

This paper introduces RadLing, a novel radiology-specific language model built on the ELECTRA architecture. By incorporating domain knowledge from RadLex and a new "knowledge-aware masking" pretraining strategy, RadLing significantly improves performance on various radiology NLP tasks, especially for underrepresented concepts.

What Problem Does This Paper Address?

Radiology reports are crucial for patient care, but extracting information from them is challenging. While general-purpose language models (PLMs) have advanced NLP, they often struggle with the specific language, context, and structure of radiology reports. Existing biomedical PLMs, trained on scientific publications, don't fully capture the nuances of radiology. Furthermore, training large PLMs is computationally expensive and requires vast amounts of data, which can be a barrier for industrial applications. This paper aims to bridge this gap by developing an efficient, domain-specific PLM that leverages existing knowledge to improve understanding of radiology reports.

Methods

The core of RadLing is the ELECTRA-small architecture, chosen for its efficiency and ability to achieve good performance with less data. The authors introduce two key innovations:

  • Domain-Specific Vocabulary Extension: They augment the standard vocabulary with radiology-specific terms identified using RadLex, a radiology lexicon. This ensures that important domain terms are treated as single tokens, preserving their meaning.
  • Knowledge-Aware Masking (RadLing-KG): This is the paper's main contribution. Instead of random masking, this strategy intelligently masks tokens based on their category (Symptom, Anatomy, Observation) and their relationship to other entities, guided by RadLex. This helps the model learn richer contextual relationships. For instance, it can learn that "pneumonia" is often associated with "lungs."
  • Section Segmentation (RadLing-SS): Another pretraining objective that helps the model understand the structure of radiology reports by classifying sentences into predefined sections (Clinical, Findings, Impressions, etc.).

These pretraining objectives are combined with a regularization term to prevent catastrophic forgetting and improve overall performance.

Datasets

  • Training Data: 545K radiology reports, comprising over 9.7 million sentences and 87 million words, collected from medical institutions in the US and India.
  • Evaluation Datasets:
    • Named Entity Recognition (NER) & Relationship Extraction (RE): RadGraph dataset (MIMIC-CXR and CheXpert subsets).
  • Abnormal Classification: A dataset from the Indiana Network for Patient Care.
  • Radiology Question Answering (RadQA): RadQA dataset.

Key Insights and Ideas

  • Domain Knowledge is Crucial: Infusing domain-specific knowledge (via RadLex) into PLMs significantly boosts performance, especially for underrepresented concepts and relationships.
  • ELECTRA's Efficiency: The ELECTRA architecture proves effective for radiology NLP, achieving strong results even with a smaller model size and less pretraining data compared to larger models.
  • Knowledge-Aware Masking Outperforms Random Masking: The novel knowledge-aware masking strategy is key to RadLing-KG's superior performance, particularly in tasks requiring nuanced understanding of entity relationships.
  • Vocabulary Matters: A domain-specific vocabulary helps preserve the meaning of critical radiology terms, preventing them from being fragmented into less meaningful sub-tokens.

Key Results

RadLing models, particularly RadLing-KG, demonstrate state-of-the-art or competitive performance across several downstream tasks:

  • NER: RadLing-KG achieved a macro F1 of 0.92 on the RadGraph test dataset, outperforming other models and significantly improving performance on underrepresented "Observation: Uncertain" entities.
  • Relationship Extraction: RadLing-KG achieved F1 scores of 0.98 and 0.94 on MIMIC and CheXpert test datasets, respectively, outperforming existing methods.
  • Abnormal Classification: RadLing-KG matched the high performance of RadBERT with a macro F1 of 0.99.
  • Question Answering: While RadLing-KG performed best among RadLing variants, it was slightly lower than the state-of-the-art RadBERT.

Limitations and Future Work

Limitations:

  • English Only: RadLing is trained on English reports and would require further adaptation for multilingual settings.
  • Data Imbalance: The training data has an imbalance in imaging modalities and anatomies, potentially limiting performance on underrepresented regions (e.g., extremities, neck).
  • Radiologist Variability: Differences in radiologist writing styles and documentation depth could impact model consistency.
  • Throughput: While efficient, the model's space and time throughput might still be a concern for edge deployments with limited bandwidth.

Future Work:

The authors suggest exploring more advanced knowledge infusion techniques, such as using text descriptions of context, retrieving information from biomedical knowledge graphs (SNOMED, UMLS), and developing more robust knowledge embedding methods. They also plan to experiment with larger datasets and models and apply RadLing to more downstream radiology applications.

Summary updated

BibTeX
@inproceedings{ghosh2023radling,
  author = {Rikhiya Ghosh and Oladimeji Farri and Sanjeev Kumar Karn and Manuela Danu and Ramya Vunikili and Larisa Micu},
  title = {{RadLing: Towards Efficient Radiology Report Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure