Simple Words over Rich Imaging: Accurate Brain Disease Classification via Language Model Analysis of Radiological Reports

Xin Gao, Meihui Zhang, Longfei Chen, Jun Qiu, Su Zhao, Junjie Li, Tiantian Hua, Ying Jin, Zhiqiang Wu, Haisong Hou, Yunling Wang, Wei Zhao, Yuxin Li, Yunyun Duan, Chuyang Ye, Yaou LiumedRxiv 2024 · DOI

TL;DR

Researchers have demonstrated that analyzing the "simple words" in radiological reports using language models is significantly more effective for classifying brain diseases than processing "rich" 3D imaging data directly. Their best-performing language model outperformed state-of-the-art image-based models by over 10% in accuracy and even surpassed junior radiologists, while proving far more resilient to missing data and differences between hospital scanning equipment.

What Problem Does This Paper Address?

Automated diagnosis of brain diseases (like identifying specific genetic mutations in tumors or grading malignancy) typically relies on deep learning models trained on 3D MRI scans. However, this "image-first" approach faces three major hurdles:

  1. High Dimensionality: 3D images with multiple contrasts are computationally expensive and complex to analyze, especially when critical lesions are tiny.
  2. Missing Data: In clinical practice, patients don't always receive a full suite of MRI sequences (e.g., missing a FLAIR or ADC contrast), which often causes image-based models to fail.
  3. Domain Shift: Models trained on images from one hospital often perform poorly at another due to different scanner brands or imaging protocols.

The authors propose that radiological reports—the text written by experts to describe these images—already contain a "concise distillation" of the most relevant information, making them a more robust and efficient data source for automated classification.

Methods

The study compares two main approaches across four clinical tasks:

  • Report-based (The Proposal): The team fine-tuned Pre-trained Language Models (PLMs) like Chinese RoBERTa and Large Language Models (LLMs) like LLaMA3-8B and Baichuan2-13B. They used the "findings" section of radiological reports as input to predict disease categories.
  • Image-based (The Baseline): They tested six state-of-the-art architectures, including 3D MedMNIST, Swin Transformers, and Vision Transformers (ViT), some of which were aided by automated tumor segmentation.

A key technical highlight was the use of LoRA (Low-Rank Adaptation) to fine-tune the LLMs efficiently on multiple tasks simultaneously, whereas the PLMs were fine-tuned for each specific task.

Datasets

The researchers used a massive collection of 12 in-house datasets involving 14,970 patients from three major Chinese hospitals:

  • Beijing Tiantan Hospital
  • The First Affiliated Hospital of Xinjiang Medical University
  • Huashan Hospital, Fudan University

The tasks included IDH genotyping, 1p/19q co-deletion identification, WHO grading, and brain tumor classification (distinguishing between Glioblastoma and Primary Central Nervous System Lymphoma). Crucially, they included independent validation sets to test how well the models generalized to "unseen" hospitals.

Key Insights and Ideas

  • Text as a "Compressed" Feature Space: While a 3D MRI contains millions of voxels, a radiological report contains a few hundred words. The radiologist has already done the "feature extraction," filtering out noise and focusing on diagnostic essentials.
  • Robustness to "Incomplete" Reality: If an MRI sequence is missing, an image-based model loses a chunk of its input. However, a radiologist can often still describe the pathology in the report based on remaining sequences. The language model benefits from this human-mediated resilience.
  • Extreme Efficiency: Pre-processing text (tokenization) was found to be 5,000 to 6,000 times faster than pre-processing 3D images (which requires skull stripping, registration, and segmentation). This makes the language model approach far more suitable for "emergency" clinical settings.
  • The "Junior Radiologist" Benchmark: The best language model achieved nearly 80% accuracy, outperforming junior radiologists with three years of experience by roughly 9.5%.

Key Results

  • Superior Accuracy: The best language model (Chinese RoBERTa) achieved an average AUC of 84.75% and Accuracy of 79.48%.
  • The Gap: Compared to the best image-based model (Swin Transformer), the language model showed a 10.75% improvement in accuracy and a 10.34% improvement in AUC.
  • Handling Missing Data: When MRI contrasts were missing, the image-based models' performance plummeted. In contrast, the language model maintained high performance, outperforming the image models by up to 32% in certain missing-data scenarios.
  • Cross-Site Generalization: On external datasets from different hospitals, the language model's accuracy was significantly more stable than the image-based models, which struggled with the "domain shift."

Limitations and Future Work

  • Sample Size: While 14,000+ patients is large, the authors suggest even larger datasets could further improve LLM performance, as LLMs typically thrive on massive scale.
  • The "Black Box" Problem: Like most deep learning models, there is a lack of "explainability." Future work should focus on why the model made a specific classification to build clinical trust.
  • PLM vs. LLM: Interestingly, the smaller, task-specific PLMs (RoBERTa) often outperformed the massive LLMs (LLaMA3). This suggests that for specific medical niche tasks, "smaller and specialized" might still beat "general and massive."
  • Language Dependency: The current study focused on Chinese reports; validating the approach across different languages and diverse reporting styles is a necessary next step.

Summary updated

Abstract

ABSTRACT Brain diseases exert profound detrimental effects on human health by affecting the central nervous system. Accurate automated diagnosis of brain diseases is imperative to delay the progression of illness and enhance long-term prognosis. However, existing image-based diagnostic approaches struggle to achieve satisfactory performance due to the high dimensionality of imaging data. Radiological reports, which are required in clinical routine to describe image findings, provide a more straightforward comprehension of the imaging data, yet they have been neglected in automated brain disease classification. In this work, we explore automated brain disease classification via radiological reports and language models and compare the results with conventional image-based methods. Specifically, in the report-based diagnostic approach, we fine-tune Pre-trained Language Models (PLMs) and Large Language Models (LLMs) based on the findings part of radiological reports to achieve disease classification. Four clinically relevant brain disease classification tasks were performed in our experiments, involving 12 datasets with a total number of 14,970 patients, including two independent validation sets. The best language model reached an average area under the receiver operating characteristic curve (AUC) of 84.75%, an average accuracy (ACC) of 79.48%, and an average F1-score of 79.45%. Compared with the best image-based model, it achieved an average improvement of 10.34%, 10.75%, and 9.95% in terms of AUC, ACC, and F1-score, respectively. The language model also outperformed junior radiologists by 9.47% in terms of ACC. Moreover, the report-based model exhibited better adaptability to missing image contrasts and cross-site data variability than image-based models. Together, these results show that brain disease classification via language model analysis of radiological reports can be more reliable than image-based classification, and our work demonstrates the potential of using radiological reports for accurate diagnosis of brain diseases.
BibTeX
@misc{gao2024simple,
  author = {Xin Gao and Meihui Zhang and Longfei Chen and Jun Qiu and Su Zhao and Junjie Li and Tiantian Hua and Ying Jin and Zhiqiang Wu and Haisong Hou and Yunling Wang and Wei Zhao and Yuxin Li and Yunyun Duan and Chuyang Ye and Yaou Liu and Yaou Liu and Yaou Liu},
  title = {{Simple Words over Rich Imaging: Accurate Brain Disease Classification via Language Model Analysis of Radiological Reports}},
  year = {2024},
  doi = {10.1101/2024.11.13.24317214},
  howpublished = {medRxiv},
}
Figure