TL;DR
This paper introduces SICAR, a lightweight multimodal language model (MLLM) for radiology report generation that achieves competitive performance with significantly fewer computational resources. It employs a novel two-stage post-processing strategy, including a "SafetyNet" with an X-ray classifier, to enhance diagnostic accuracy and patient safety, making advanced AI more accessible to resource-constrained hospitals.
What Problem Does This Paper Address?
Radiology report generation (RRG) is a critical but time-consuming task for radiologists, leading to potential bottlenecks and inconsistencies. While AI has shown promise in automating this process, existing solutions often require substantial computational power (high-end GPUs) and raise privacy concerns due to cloud-based deployment. This is particularly challenging for hospitals in developing countries or those with limited IT infrastructure. Furthermore, ensuring the accuracy and safety of AI-generated reports, especially avoiding critical false negatives, remains a significant hurdle.
Methods
The authors propose SICAR, a system built around a lightweight MLLM. The core architecture consists of:
- Visual Encoder: SigLIP-so400m, a general-domain model for extracting features from chest X-ray images.
- Vision-Language Connector: A two-layer Multi-layer Perceptron (MLP) with GELU activation.
- Language Model: Phi-2 2.7B, a compact yet powerful LLM.
The training involves a two-stage strategy:
- Stage 1 (MLP Connector Training): The MLP connector is trained on the LLaVa-Med alignment 500k dataset, with the visual encoder and LLM frozen. This stage focuses on mapping medical visual features to language representations.
- Stage 2 (Fine-tuning): Both the vision-language connector and the LLM are fine-tuned on the interpret-cxr dataset, which comprises multiple chest X-ray datasets. The visual encoder remains frozen.
A key innovation is the two-stage post-processing strategy:
- Report Refinement: An LLM is used to improve the readability, clarity, and conciseness of the generated reports, standardizing vocabulary and transforming simple statements into detailed descriptions.
- "First, Do No Harm" SafetyNet: This stage integrates X-Raydar, a state-of-the-art X-ray classification model. X-Raydar acts as a second opinion, cross-verifying the MLLM's outputs to specifically catch and correct false negative errors, thereby enhancing diagnostic accuracy and patient safety.
Datasets
- LLaVa-Med alignment 500k: Used for pre-training the MLP connector.
- interpret-cxr: A combined dataset including CheXpert, PadChest, BIMCV COVID-19, and MIMIC-CXR-JPG. This was used for fine-tuning the MLLM.
- Public Findings Benchmark: Used for evaluating various LLM and visual encoder combinations.
- Hidden Findings Benchmark: Used to evaluate the performance improvement of the post-processing stages.
Key Insights and Ideas
- Lightweight models can compete: The study demonstrates that a smaller LLM like Phi-2 2.7B, when combined with a suitable visual encoder and a robust training/post-processing pipeline, can achieve performance comparable to larger models. This is crucial for enabling on-premise deployment on consumer-grade GPUs.
- The power of two-stage post-processing: The sequential refinement and "SafetyNet" approach significantly boosts report quality and, more importantly, clinical accuracy. The SafetyNet, by incorporating an external classifier, acts as a critical safeguard against dangerous errors.
- Visual encoder quality matters: The findings suggest that the effectiveness of the visual encoder in extracting relevant information from X-ray images might be more critical for this specific task than the sheer size or pre-trained knowledge of the language model.
- Data selection impacts performance: Using only the first image from a study, rather than all available images, yielded slightly better results, possibly by reducing redundancy and bias.
Key Results
- The proposed MLLM architecture (Phi-2 + SigLIP) achieved competitive performance on the F1-Radgraph metric for findings generation, securing fourth place in the RRG2024 shared task.
- The two-stage post-processing strategy led to a significant improvement in the F1-Radgraph metric, increasing it by 2.01 points (from 22.61 to 24.62) over the base model.
- The "First, Do No Harm" SafetyNet stage alone contributed a 1.51-point increase in the F1-Radgraph metric.
- The authors found that initializing with a general-domain MLLM (like Bunny) or an LLM with a randomly initialized adapter performed better than initializing solely from a general LLM.
Limitations and Future Work
- LLM size for post-processing: The study used Llama3 70B for post-processing but did not explicitly analyze the impact of smaller LLMs on this stage's performance. Future work could explore this to further optimize resource usage.
- Sequential post-processing: The current two-stage post-processing might not fully leverage LLM capabilities and could introduce latency. Combining these stages into a single step is suggested for improved efficiency.
- Dataset cleaning impact: While GPT-3.5 Turbo was used for dataset cleaning, it resulted in a slight performance decline, possibly due to inconsistencies with public test sets. Further investigation into optimal data cleaning strategies is warranted.
Summary updated