TL;DR
While multimodal LLMs like GPT-4(V) can "see" medical images, their diagnostic accuracy in complex brain MRI cases depends almost entirely on the quality of the accompanying text. This study reveals that providing a detailed textual description of radiological findings is the single most important factor for success, while the images themselves—even when annotated with arrows—contribute surprisingly little to the final diagnosis.
What Problem Does This Paper Address?
The "Vision" in GPT-4(V) has sparked hope for automated radiological assistance, but early tests have shown inconsistent results. The core challenge is understanding the "black box" of multimodal prompting: when a model gets a diagnosis right, is it because it "saw" the pathology in the pixels, or because it inferred it from the patient’s clinical history or the way the findings were described?
In neuroradiology, where findings are often subtle and highly complex, identifying which prompt elements (images, annotations, history, or descriptions) actually drive diagnostic performance is crucial for developing reliable clinical AI tools.
Methods
The researchers selected 30 challenging brain MRI cases with verified diagnoses (via histopathology or senior expert consensus). They then tested GPT-4(V) across seven different prompt groups to isolate the impact of four specific input elements:
- Image (I): Representative MRI slices.
- Annotation (A): Arrows pointing to key findings on the images.
- Medical History (H): Clinical context from the patient record.
- Image Description (D): A standardized textual summary of location, morphology, and signal behavior.
Using PerplexityAI (powered by GPT-4(V)), they ran each case through each prompt group three times to account for LLM variability. Accuracy was measured using a binary system (was the correct diagnosis in the top 3?) and a numeric rank-based score (0–3). Finally, they used mixed-effects regression models to statistically "tease apart" how much each individual element contributed to the model's success.
Datasets
- Primary Dataset: 30 complex brain MRI cases from the local database of the Technical University of Munich.
- Verification: All cases had definitive "ground truth" diagnoses confirmed by biopsy or independent agreement between two senior neuroradiologists.
- Availability: 20 of these cases had been previously published in medical literature, representing a high level of diagnostic difficulty.
Key Insights and Ideas
- The "Vision Gap": The most striking insight is that GPT-4(V) is currently a much better "reasoner" than it is a "see-er" in a medical context. It struggles to extract high-level diagnostic features directly from raw MRI pixels.
- Text is the Primary Driver: The textual description of findings (D) was the "super-factor." Without it, accuracy plummeted. This suggests the model relies on the expert's translation of visual data into semantic concepts.
- Annotations are Counter-Intuitive: Adding arrows (annotations) to images did not significantly help and, in some regression models, showed a minor negative effect. This suggests that visual markers might introduce noise or "distract" the model's attention mechanism rather than clarifying the pathology.
- Clinical Context Matters: Medical history (H) provided a moderate, statistically significant boost, confirming that the model uses clinical logic to narrow down possibilities, much like a human radiologist.
Key Results
- Peak Performance: The "all-in" prompt (Image + Annotation + History + Description) achieved the highest accuracy at 67.8%.
- The Failure of "Vision-Only": Prompts containing only the image (4.4% accuracy) or image + annotations (2.2% accuracy) performed abysmally.
- The Description Effect: Any prompt group that included a textual description (Groups 4–7) performed significantly better than those that did not (Groups 1–3).
- Statistical Weight: Regression analysis showed the Image Description (D) had a coefficient of 4.420 (p << 0.001), dwarfing the impact of Medical History (1.196) and the Image itself (0.761).
Limitations and Future Work
- Generalist vs. Specialist: GPT-4(V) is a general-purpose model. The authors note that domain-specific models trained specifically on medical imaging might rely less on text and more on visual data.
- Complexity Bias: The study used "challenging" cases. The model might perform better on "textbook" cases with only images, though this remains to be proven.
- Description Bias: The descriptions were written by experts who knew the diagnosis. While they avoided using the diagnosis name, their choice of descriptive terminology might have inadvertently "steered" the model.
- 2D vs. 3D: The study used 2D slices. Future work should explore whether providing 3D volumes or more comprehensive slice sets improves the model's native visual processing.
- The "Two-Step" Workflow: A promising future direction is investigating if an LLM can first generate its own description of an image and then use that description to self-diagnose in a second step.
Summary updated