TL;DR
Aggregate accuracy metrics in medical Vision-Language Models (VLMs) can mask catastrophic, diagnosis-specific failures. This paper demonstrates that several state-of-the-art models exhibit "silent blindness"—confidently denying the presence of life-threatening conditions like pneumothorax even while fluently describing their radiographic signs—despite the information being present in the models' internal representations.
What Problem Does This Paper Address?
In medical AI, we often rely on a single "headline" accuracy number to judge a model's performance. However, an average score can be deceptive. A model might achieve 90% accuracy by being perfect at identifying common conditions while completely failing to detect a rare but fatal one.
The authors investigate a phenomenon they call "whole-diagnosis collapse." They focus specifically on pneumothorax (a collapsed lung), a high-stakes diagnosis where a missed case can be fatal within hours. The danger here is twofold: not only do some models fail to see the condition, but they do so "silently." They provide fluent, medically plausible explanations for why the condition is absent, making the error incredibly difficult for a clinician to catch at the point of care.
Methods
The researchers audited five open-source medical VLMs: MedGemma-4B, Lingshu-7B, HuatuoGPT-Vision-7B, Qwen2.5-VL-7B, and MedVLM-R1.
Their methodology included several rigorous controls:
- Prevalence-Matched Audit: They constructed a balanced dataset to ensure that high accuracy wasn't just a result of the model guessing the most common label.
- Grounding Metric: They tested if model answers were actually tied to the image by comparing responses to original images versus "swapped" images (different patient, opposite label) and "augmented" images (slight rotations or brightness changes).
- Linear Probing: To see if the models were truly "blind," they trained simple classifiers (linear probes) on the models' internal hidden states. This allowed them to check if the information about a diagnosis existed inside the model, even if the final text output was a "No."
- Robustness Checks: They tested whether failures were due to prompt wording, image resolution, or model size (scaling up to 72B parameters).
Datasets
- ChestX-ray14: The primary dataset used for the audit, consisting of over 100,000 frontal X-rays. The authors used a curated subset of 6,048 items balanced by age, sex, and diagnosis.
- SIIM-ACR Pneumothorax: An independent validation set featuring radiologist-drawn, pixel-level masks. This was used to ensure the findings weren't just an artifact of noisy labels in the primary dataset.
Key Insights and Ideas
- Fluent Hallucination of Absence: The most striking finding is that failing models don't just say "No"; they explain why in detail. As shown in Figure 2, a model might correctly state that a pneumothorax appears as a "clear space between the lung and chest wall" and then immediately claim that space is not present, even when it is clearly visible.
- Internal Representation vs. Output Suppression: The linear probe analysis revealed that the models actually "see" the pneumothorax. The information is decodable from early transformer layers but "fades" or is suppressed before reaching the final output layer. This suggests the failure is a logic or language-generation issue rather than a purely perceptual one.
- Scale is Not a Panacea: Increasing the model size (e.g., Qwen2.5-VL from 7B to 72B) did not fix the pneumothorax blindness. This indicates that simply throwing more parameters at the problem won't necessarily solve specific diagnostic failures.
- Fine-tuning is a Double-Edged Sword: Lingshu and HuatuoGPT share the same Qwen backbone. However, Lingshu (a medical fine-tune) succeeded where HuatuoGPT (another medical fine-tune) failed completely. This highlights how specific training data and alignment choices can inadvertently "break" a model's ability to report certain findings.
Key Results
- Zero Recall: Both Qwen2.5-VL and HuatuoGPT-Vision detected zero out of 432 pneumothorax cases, effectively performing at chance level (0.500 balanced accuracy) by always answering "No."
- Diagnosis-Specific Collapse: As shown in Table 2 (Figure 4), models that were excellent at one finding (e.g., MedGemma's 94.7% recall on Atelectasis) could be significantly worse at others (37.7% on Pneumothorax).
- Internal Knowledge: Despite 0% recall at the output, linear probes achieved an AUC of 0.72 to 0.75 on the SIIM-ACR dataset for the failing models. The information was there; the model just wouldn't say it.
- Confidence vs. Accuracy: The failing models were "confidently wrong." Qwen, for instance, was 97.6% confident in its pneumothorax answers while being no better than a coin flip in reality.
Limitations and Future Work
The authors acknowledge several limitations:
- Label Noise: ChestX-ray14 labels are derived from radiology reports using NLP, which can be imperfect. While the SIIM-ACR replication mitigated this, it remains a factor in large-scale medical AI research.
- Scope: The study focused on binary "Yes/No" questions rather than the complex, free-form reports radiologists actually write.
- Demographics: While they balanced for age and sex, other demographic factors (like race or socioeconomic status) were not available in these datasets.
Future Work: The authors suggest that because the information is present internally, the failure might be addressable at the output stage (e.g., through better alignment or targeted interventions) rather than requiring a complete overhaul of the model's vision system. They also call for a shift in how we evaluate medical AI: moving away from aggregate scores toward mandatory per-finding reporting on balanced datasets.
Summary updated