Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

Severin Schramm, Silas Preis, Marie‐Christin Metz, Kirsten Jung, Benita Schmitz‐Koep, Claus Zimmer, Benedikt Wiestler, Dennis M. Hedderich, Su Hwan KimmedRxiv 2024 · DOI

TL;DR

While multimodal LLMs like GPT-4(V) can "see" medical images, their diagnostic accuracy in complex brain MRI cases depends almost entirely on the quality of the accompanying text. This study reveals that providing a detailed textual description of radiological findings is the single most important factor for success, while the images themselves—even when annotated with arrows—contribute surprisingly little to the final diagnosis.

What Problem Does This Paper Address?

The "Vision" in GPT-4(V) has sparked hope for automated radiological assistance, but early tests have shown inconsistent results. The core challenge is understanding the "black box" of multimodal prompting: when a model gets a diagnosis right, is it because it "saw" the pathology in the pixels, or because it inferred it from the patient’s clinical history or the way the findings were described?

In neuroradiology, where findings are often subtle and highly complex, identifying which prompt elements (images, annotations, history, or descriptions) actually drive diagnostic performance is crucial for developing reliable clinical AI tools.

Methods

The researchers selected 30 challenging brain MRI cases with verified diagnoses (via histopathology or senior expert consensus). They then tested GPT-4(V) across seven different prompt groups to isolate the impact of four specific input elements:

  1. Image (I): Representative MRI slices.
  2. Annotation (A): Arrows pointing to key findings on the images.
  3. Medical History (H): Clinical context from the patient record.
  4. Image Description (D): A standardized textual summary of location, morphology, and signal behavior.

Using PerplexityAI (powered by GPT-4(V)), they ran each case through each prompt group three times to account for LLM variability. Accuracy was measured using a binary system (was the correct diagnosis in the top 3?) and a numeric rank-based score (0–3). Finally, they used mixed-effects regression models to statistically "tease apart" how much each individual element contributed to the model's success.

Datasets

  • Primary Dataset: 30 complex brain MRI cases from the local database of the Technical University of Munich.
  • Verification: All cases had definitive "ground truth" diagnoses confirmed by biopsy or independent agreement between two senior neuroradiologists.
  • Availability: 20 of these cases had been previously published in medical literature, representing a high level of diagnostic difficulty.

Key Insights and Ideas

  • The "Vision Gap": The most striking insight is that GPT-4(V) is currently a much better "reasoner" than it is a "see-er" in a medical context. It struggles to extract high-level diagnostic features directly from raw MRI pixels.
  • Text is the Primary Driver: The textual description of findings (D) was the "super-factor." Without it, accuracy plummeted. This suggests the model relies on the expert's translation of visual data into semantic concepts.
  • Annotations are Counter-Intuitive: Adding arrows (annotations) to images did not significantly help and, in some regression models, showed a minor negative effect. This suggests that visual markers might introduce noise or "distract" the model's attention mechanism rather than clarifying the pathology.
  • Clinical Context Matters: Medical history (H) provided a moderate, statistically significant boost, confirming that the model uses clinical logic to narrow down possibilities, much like a human radiologist.

Key Results

  • Peak Performance: The "all-in" prompt (Image + Annotation + History + Description) achieved the highest accuracy at 67.8%.
  • The Failure of "Vision-Only": Prompts containing only the image (4.4% accuracy) or image + annotations (2.2% accuracy) performed abysmally.
  • The Description Effect: Any prompt group that included a textual description (Groups 4–7) performed significantly better than those that did not (Groups 1–3).
  • Statistical Weight: Regression analysis showed the Image Description (D) had a coefficient of 4.420 (p << 0.001), dwarfing the impact of Medical History (1.196) and the Image itself (0.761).

Limitations and Future Work

  • Generalist vs. Specialist: GPT-4(V) is a general-purpose model. The authors note that domain-specific models trained specifically on medical imaging might rely less on text and more on visual data.
  • Complexity Bias: The study used "challenging" cases. The model might perform better on "textbook" cases with only images, though this remains to be proven.
  • Description Bias: The descriptions were written by experts who knew the diagnosis. While they avoided using the diagnosis name, their choice of descriptive terminology might have inadvertently "steered" the model.
  • 2D vs. 3D: The study used 2D slices. Future work should explore whether providing 3D volumes or more comprehensive slice sets improves the model's native visual processing.
  • The "Two-Step" Workflow: A promising future direction is investigating if an LLM can first generate its own description of an image and then use that description to self-diagnose in a second step.

Summary updated

Abstract

Abstract Background Recent studies have explored the application of multimodal large language models (LLMs) in radiological differential diagnosis. Yet, how different multimodal input combinations affect diagnostic performance is not well understood. Purpose To evaluate the impact of varying multimodal input elements on the accuracy of GPT-4(V)-based brain MRI differential diagnosis. Methods Thirty brain MRI cases with a challenging yet verified diagnosis were selected. Seven prompt groups with variations of four input elements (image, image annotation, medical history, image description) were defined. For each MRI case and prompt group, three identical queries were performed using an LLM-based search engine (© PerplexityAI, powered by GPT-4(V)). Accuracy of LLM-generated differential diagnoses was rated using a binary and a numeric scoring system and analyzed using a chi-square test and a Kruskal-Wallis test. Results were corrected for false discovery rate employing the Benjamini-Hochberg procedure. Regression analyses were performed to determine the contribution of each individual input element to diagnostic performance. Results The prompt group containing an annotated image, medical history, and image description as input exhibited the highest diagnostic accuracy (67.8% correct responses). Significant differences were observed between prompt groups, especially between groups that contained the image description among their inputs, and those that did not. Regression analyses confirmed a large positive effect of the image description on diagnostic accuracy (p ≪ 0.001), as well as a moderate positive effect of the medical history (p &lt; 0.001). The presence of unannotated or annotated images had only minor or insignificant effects on diagnostic accuracy. Conclusion The textual description of radiological image findings was identified as the strongest contributor to performance of GPT-4(V) in brain MRI differential diagnosis, followed by the medical history. The unannotated or annotated image alone yielded very low diagnostic performance. These findings offer guidance on the effective utilization of multimodal LLMs in clinical practice.
BibTeX
@misc{schramm2024impact,
  author = {Severin Schramm and Silas Preis and Marie‐Christin Metz and Kirsten Jung and Benita Schmitz‐Koep and Claus Zimmer and Benedikt Wiestler and Dennis M. Hedderich and Su Hwan Kim},
  title = {{Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases}},
  year = {2024},
  doi = {10.1101/2024.03.05.24303767},
  howpublished = {medRxiv},
}
Figure