SICAR at RRG2024: GPU Poor’s Guide to Radiology Report Generation

Kiartnarin Udomlapsakul, Parinthapat Pengpun, Tossaporn Saengja, Kanyakorn Veerakanjana, Krittamate Tiankanon, Pitikorn Khlaisamniang, Pasit Supholkhan, Amrest Chinkamol, Pubordee Aussavavirojekul, Hirunkul Phimsiri, Tara Sripo, Chiraphat Boonnag, Trongtum Tongdee, Thanongchai Siriapisith, Pairash Saiviroonporn, Jiramet Kinchagawat, Piyalitt IttichaiwongACL 2024

TL;DR

This paper introduces SICAR, a lightweight multimodal language model (MLLM) for radiology report generation that achieves competitive performance with significantly fewer computational resources. It employs a novel two-stage post-processing strategy, including a "SafetyNet" with an X-ray classifier, to enhance diagnostic accuracy and patient safety, making advanced AI more accessible to resource-constrained hospitals.

What Problem Does This Paper Address?

Radiology report generation (RRG) is a critical but time-consuming task for radiologists, leading to potential bottlenecks and inconsistencies. While AI has shown promise in automating this process, existing solutions often require substantial computational power (high-end GPUs) and raise privacy concerns due to cloud-based deployment. This is particularly challenging for hospitals in developing countries or those with limited IT infrastructure. Furthermore, ensuring the accuracy and safety of AI-generated reports, especially avoiding critical false negatives, remains a significant hurdle.

Methods

The authors propose SICAR, a system built around a lightweight MLLM. The core architecture consists of:

  • Visual Encoder: SigLIP-so400m, a general-domain model for extracting features from chest X-ray images.
  • Vision-Language Connector: A two-layer Multi-layer Perceptron (MLP) with GELU activation.
  • Language Model: Phi-2 2.7B, a compact yet powerful LLM.

The training involves a two-stage strategy:

  1. Stage 1 (MLP Connector Training): The MLP connector is trained on the LLaVa-Med alignment 500k dataset, with the visual encoder and LLM frozen. This stage focuses on mapping medical visual features to language representations.
  2. Stage 2 (Fine-tuning): Both the vision-language connector and the LLM are fine-tuned on the interpret-cxr dataset, which comprises multiple chest X-ray datasets. The visual encoder remains frozen.

A key innovation is the two-stage post-processing strategy:

  1. Report Refinement: An LLM is used to improve the readability, clarity, and conciseness of the generated reports, standardizing vocabulary and transforming simple statements into detailed descriptions.
  2. "First, Do No Harm" SafetyNet: This stage integrates X-Raydar, a state-of-the-art X-ray classification model. X-Raydar acts as a second opinion, cross-verifying the MLLM's outputs to specifically catch and correct false negative errors, thereby enhancing diagnostic accuracy and patient safety.

Datasets

  • LLaVa-Med alignment 500k: Used for pre-training the MLP connector.
  • interpret-cxr: A combined dataset including CheXpert, PadChest, BIMCV COVID-19, and MIMIC-CXR-JPG. This was used for fine-tuning the MLLM.
  • Public Findings Benchmark: Used for evaluating various LLM and visual encoder combinations.
  • Hidden Findings Benchmark: Used to evaluate the performance improvement of the post-processing stages.

Key Insights and Ideas

  • Lightweight models can compete: The study demonstrates that a smaller LLM like Phi-2 2.7B, when combined with a suitable visual encoder and a robust training/post-processing pipeline, can achieve performance comparable to larger models. This is crucial for enabling on-premise deployment on consumer-grade GPUs.
  • The power of two-stage post-processing: The sequential refinement and "SafetyNet" approach significantly boosts report quality and, more importantly, clinical accuracy. The SafetyNet, by incorporating an external classifier, acts as a critical safeguard against dangerous errors.
  • Visual encoder quality matters: The findings suggest that the effectiveness of the visual encoder in extracting relevant information from X-ray images might be more critical for this specific task than the sheer size or pre-trained knowledge of the language model.
  • Data selection impacts performance: Using only the first image from a study, rather than all available images, yielded slightly better results, possibly by reducing redundancy and bias.

Key Results

  • The proposed MLLM architecture (Phi-2 + SigLIP) achieved competitive performance on the F1-Radgraph metric for findings generation, securing fourth place in the RRG2024 shared task.
  • The two-stage post-processing strategy led to a significant improvement in the F1-Radgraph metric, increasing it by 2.01 points (from 22.61 to 24.62) over the base model.
  • The "First, Do No Harm" SafetyNet stage alone contributed a 1.51-point increase in the F1-Radgraph metric.
  • The authors found that initializing with a general-domain MLLM (like Bunny) or an LLM with a randomly initialized adapter performed better than initializing solely from a general LLM.

Limitations and Future Work

  • LLM size for post-processing: The study used Llama3 70B for post-processing but did not explicitly analyze the impact of smaller LLMs on this stage's performance. Future work could explore this to further optimize resource usage.
  • Sequential post-processing: The current two-stage post-processing might not fully leverage LLM capabilities and could introduce latency. Combining these stages into a single step is suggested for improved efficiency.
  • Dataset cleaning impact: While GPT-3.5 Turbo was used for dataset cleaning, it resulted in a slight performance decline, possibly due to inconsistencies with public test sets. Further investigation into optimal data cleaning strategies is warranted.

Summary updated

BibTeX
@inproceedings{udomlapsakul2024sicar,
  author = {Kiartnarin Udomlapsakul and Parinthapat Pengpun and Tossaporn Saengja and Kanyakorn Veerakanjana and Krittamate Tiankanon and Pitikorn Khlaisamniang and Pasit Supholkhan and Amrest Chinkamol and Pubordee Aussavavirojekul and Hirunkul Phimsiri and Tara Sripo and Chiraphat Boonnag and Trongtum Tongdee and Thanongchai Siriapisith and Pairash Saiviroonporn and Jiramet Kinchagawat and Piyalitt Ittichaiwong},
  title = {{SICAR at RRG2024: GPU Poor’s Guide to Radiology Report Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure