Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting

Ryan Shea Ying Cong Tan, Qian Lin, Guat Hwa Low, Ruixi Lin, Tzer Chew Goh, Christopher Chu En Chang, Fung Fung Lee, Wei Yin Chan, Wei Chong Tan, Han Jieh Tey, Fun Loon Leong, Hong Qi Tan, Wen Long Nei, Wen Yee Chay, David Wai Meng Tai, Gillianne Geet Yi Lai, Lionel Tim-Ee Cheng, Fuh Yong Wong, Matthew Chin Heng Chua, Melvin Lee Kiang Chua, Daniel Shao-Weng Tan, Choon Hua Thng, Iain Bee Huat Tan, Hwee Tou NgJ. Am. Medical Informatics Assoc 2023 · OpenReview

TL;DR

This paper explores how large language models (LLMs) can automatically infer cancer disease response from free-text radiology reports, achieving high accuracy. They demonstrate that data augmentation improves performance, while prompt-based fine-tuning significantly reduces the need for large training datasets, making LLMs more accessible for clinical research and decision support.

What Problem Does This Paper Address?

The vast amount of clinical data, particularly in oncology, is often locked away in unstructured free-text formats like radiology reports. Extracting critical information, such as a patient's cancer disease response, from these reports is crucial for advancing research, improving patient care, and enabling real-world evidence studies. Traditionally, this extraction has required manual review by human annotators, which is time-consuming, expensive, and prone to variability. This paper aims to bridge this gap by leveraging the power of LLMs to automate this process, making it faster, more scalable, and potentially more consistent.

Methods

The researchers employed a variety of natural language processing (NLP) techniques to classify cancer disease response from radiology reports. They experimented with:

  • Transformer Models: A suite of advanced LLMs, including BERT, BioBERT, GatorTron, and others, were evaluated. These models are known for their ability to understand context and relationships within text.
  • Conventional Models: For comparison, they also used a bidirectional long short-term memory (Bi-LSTM) model, a convolutional neural network (CNN), and traditional machine learning methods like linear support vector machines (linearSVC) and ensemble voting classifiers.
  • Data Augmentation: To improve model robustness and performance, they used a technique called "sentence permutation with consistency loss." This involves shuffling sentences within a report's conclusion to create synthetic training data, while ensuring the original disease response label remains unchanged.
  • Prompt-Based Fine-Tuning: They explored ADAPET, a method that uses pre-trained language models with specific "cloze-style" prompts (e.g., "In summary, this is a [MASK]"). This approach aims to guide the LLM towards the desired classification task and potentially reduce the amount of labeled data needed for effective training.

Datasets

The study utilized a substantial dataset of deidentified computed tomography (CT) radiology reports from cancer patients at the National Cancer Centre Singapore (NCCS).

  • Internal Dataset: 10,602 CT reports from 1740 patients were collected, covering colorectal, lung, breast, and gynaecological cancers. These reports were manually curated and classified into four categories: no evidence of disease (NED), partial response (PR), stable disease (SD), or progressive disease (PD).
  • External Validation Dataset: An independent set of 37 radiology reports with accompanying RECIST (Response Evaluation Criteria in Solid Tumors) classifications was used to validate the best-performing model.

Key Insights and Ideas

  • LLMs Excel at Clinical Text: The study strongly supports the idea that transformer-based LLMs are significantly better at inferring cancer disease response from radiology reports compared to older NLP methods.
  • Data Augmentation is Beneficial: Even with a large initial dataset, sentence permutation can further boost the accuracy of LLMs, highlighting its utility in enhancing model performance.
  • Prompting for Efficiency: Prompt-based fine-tuning, while not always improving accuracy over traditional fine-tuning on large datasets, dramatically reduces the amount of labeled data required to achieve good performance. This is a critical finding for real-world applications where obtaining large, annotated datasets is challenging.
  • Interpretability Matters: Visualizing model attention weights (as shown in Figures 2 and 3) offers a glimpse into why the model makes certain predictions, which is crucial for building trust and understanding in clinical settings.

Key Results

  • Top Performer: The GatorTron transformer model achieved the highest accuracy, reaching 0.8916 on the internal test set and 0.8919 on the external RECIST validation set.
  • Data Augmentation Boost: Data augmentation further improved GatorTron's accuracy to 0.8976.
  • Prompting Efficiency: Prompt-based fine-tuning with GatorTron achieved a good accuracy of 0.8609 using only 500 training reports, a significant reduction from the full training set. This also reduced training time by 63% when using the full dataset.
  • Inter-Annotator Agreement: Human annotators showed high agreement, with a Cohen's kappa statistic of 0.83, indicating a reliable gold standard for model evaluation.

Limitations and Future Work

  • Data Specificity: The study's radiology reports were restricted to those involving multiple anatomical regions and originated from a single institution. The generalizability of the model to other hospitals and different types of reports remains to be seen.
  • External Dataset Size: The external RECIST validation dataset was relatively small (37 reports), and further studies are needed to validate these findings.
  • Future Directions: The authors suggest that these models could be valuable for researchers to derive progression-free survival in large datasets and serve as a clinical decision support tool, providing clinicians with an automated second opinion. They also highlight the potential for using these NLP models in clinical trials to improve trial conduct efficiency.

Summary updated

BibTeX
@article{tan2023inferring,
  author = {Ryan Shea Ying Cong Tan and Qian Lin and Guat Hwa Low and Ruixi Lin and Tzer Chew Goh and Christopher Chu En Chang and Fung Fung Lee and Wei Yin Chan and Wei Chong Tan and Han Jieh Tey and Fun Loon Leong and Hong Qi Tan and Wen Long Nei and Wen Yee Chay and David Wai Meng Tai and Gillianne Geet Yi Lai and Lionel Tim-Ee Cheng and Fuh Yong Wong and Matthew Chin Heng Chua and Melvin Lee Kiang Chua and Daniel Shao-Weng Tan and Choon Hua Thng and Iain Bee Huat Tan and Hwee Tou Ng},
  title = {{Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2023},
  url = {https://openreview.net/forum?id=W7TGwF4Q0bw},
}
Figure