TL;DR
This paper introduces "GPTRadScore," a novel framework that uses GPT-4 to automatically evaluate how well multimodal Large Language Models (LLMs) describe findings in CT scans, mimicking human radiologist assessments. The study reveals that while leading models like GPT-4V and Gemini Pro Vision show promise, they still have significant room for improvement, especially in accurately identifying lesion locations. Fine-tuning models on domain-specific data, as demonstrated with RadFM, can substantially boost their performance.
What Problem Does This Paper Address?
Radiologists face an ever-increasing workload, partly due to the growing volume of CT scans. Automatically generating descriptions of findings in these scans could significantly ease this burden. However, evaluating the accuracy and clinical relevance of these AI-generated descriptions is challenging. Existing metrics like BLEU and ROUGE, which focus on word overlap, fail to capture the nuanced understanding required for medical reports. Furthermore, there's a scarcity of large, annotated datasets and robust evaluation frameworks that truly reflect clinical utility. This paper aims to bridge this gap by proposing a new evaluation method and assessing current multimodal LLMs' capabilities in this critical domain.
Methods
The core of this work is the GPTRadScore framework. It works by:
- Generating Descriptions: Multimodal LLMs (GPT-4V, Gemini Pro Vision, LLaVA-Med, RadFM) are fed CT slices with identified abnormalities (marked by bounding boxes for visual context). They then generate free-text descriptions of the findings.
- Decomposition and Evaluation: GPT-4 is then used to analyze these generated descriptions. It breaks them down into key components: body part, location, and type of finding. GPT-4 then compares these components against gold-standard, clinician-annotated sentences, assigning scores for accuracy and clinical relevance. This mimics how a radiologist would evaluate a report.
- Fine-tuning: To improve performance, the RadFM model was fine-tuned on a subset of the DeepLesion dataset, specifically curated for complex findings.
Datasets
- DeepLesion Dataset: This retrospective study utilized a subset of the DeepLesion dataset, which contains CT slices and corresponding radiology reports. Specifically, they used 500 CT slices with prospectively marked lesions and extracted characteristics (body part, location, type) from the reports.
- New Benchmark: The authors plan to release a benchmark dataset annotated by clinicians to facilitate future research.
Key Insights and Ideas
- LLMs as Evaluators: Using GPT-4 to evaluate the output of other LLMs is a powerful approach. It leverages GPT-4's advanced reasoning capabilities to mimic expert judgment, offering a more nuanced assessment than traditional NLP metrics.
- Decomposition is Key: Breaking down the evaluation into specific components (body part, location, type) allows for a more granular understanding of where LLMs excel and where they struggle.
- Visual Cues Matter: Bounding boxes highlighting lesions significantly improve the accuracy of LLMs, especially for identifying body parts and locations, underscoring the importance of clear visual context.
- Chain-of-Thought (CoT) Benefits: While CoT reasoning helps LLMs produce more detailed and relevant descriptions, its effectiveness varies. It particularly benefits complex tasks requiring nuanced interpretation, but can sometimes hinder performance on simpler tasks like location identification.
- Fine-tuning is Crucial: Models not specifically trained for medical imaging tasks (like RadFM) show significant performance gains when fine-tuned on domain-specific data, highlighting the need for specialized training.
Key Results
- GPTRadScore Correlation: The GPTRadScore framework showed a high correlation (Pearson's correlation coefficient) with clinician assessments, validating its reliability as an evaluation metric.
- Model Performance: GPT-4V and Gemini Pro Vision generally outperformed other models, but still had limitations, particularly in lesion location accuracy (e.g., GPT-4V at 17.1%, Gemini Pro Vision at 14%).
- RadFM Improvement: Fine-tuning RadFM led to substantial accuracy increases: location accuracy rose from 3.41% to 12.8%, body part accuracy from 29.12% to 53%, and type accuracy from 9.24% to 30%.
- Bounding Box Impact: Using bounding boxes consistently improved accuracy across models, especially for body part and location identification.
- Traditional Metrics Fall Short: Traditional NLG metrics like BLEU and ROUGE showed weak correlations with ground truth and clinician evaluations, highlighting their inadequacy for assessing clinical relevance in radiology reports.
Limitations and Future Work
- Prompt Engineering: The study used prompts recommended by model developers. More meticulously crafted prompts could potentially yield better results, suggesting future work in automated prompt generation or optimization.
- Cost and Practicality: Implementing advanced AI models in clinical settings is costly. Future work could focus on developing computationally efficient models, cloud-based deployments, and partnerships to mitigate these costs.
- Dataset Limitations: The performance of LLMs is significantly impacted by the limitations of their training datasets. Future research should focus on creating larger, more diverse, and expertly annotated datasets.
- Generalization: Models like LLaVA-Med and RadFM struggled without spatial cues, indicating challenges in generalizing from training data. Further research is needed to improve their adaptability to real-world medical tasks.
Summary updated