Evaluating Cross-Signal Coherence in Mammography Vision-Language Models

Rebeca Nam, Chase M. Fensore, Christian A. Fauria-Robinson, Judy Gichoya, Joyce C. HoarXiv 2026 · arXiv

This paper investigates the "black box" of Vision-Language Models (VLMs) in the high-stakes domain of mammography. The authors move beyond simple accuracy to evaluate coherence: do a model's answer, its stated confidence, and its written explanation actually align, and are they grounded in visual evidence?

TL;DR

Researchers found that mammography VLMs often exhibit "mirage reasoning," where they provide confident, detailed explanations that contradict their final answers or persist even when the medical image is removed. The study reveals that these models are highly sensitive to superficial changes like option ordering, suggesting that current medical VLM performance is often anchored in linguistic patterns rather than genuine clinical understanding.

What Problem Does This Paper Address?

As VLMs are increasingly proposed for autonomous medical tasks—like triaging mammograms—evaluators typically rely on accuracy metrics. However, in medicine, being "right for the wrong reasons" is dangerous. A model might correctly label a case as BI-RADS 0 (needs further imaging) but its explanation might describe the breast as "normal" (which should be BI-RADS 1).

The authors address the lack of multi-signal evaluation. They argue that for a clinician to trust an AI, the model’s three primary outputs—the answer, the confidence score, and the rationale—must be internally consistent and externally grounded in the image data.

Methods

The researchers evaluated four open-source VLMs: two medically-adapted (MedGemma-4B, Lingshu-7B) and two general-purpose (Qwen2-VL-7B, InternVL3-8B).

They employed a novel evaluation framework involving:

  1. Joint Signal Elicitation: Using a structured prompt to force the model to provide an answer, a confidence score for every option (not just the chosen one), and a one-sentence explanation.
  2. Controlled Perturbations: To test visual grounding and elicitation robustness, they subjected the models to six conditions:
  3. Original: The baseline image and question.
  4. Visual Perturbations: Replacing the mammogram with a blank image, an unrelated photo (e.g., a pastry), or removing the image entirely.
  5. Linguistic Perturbations: Shuffling the order of multiple-choice options and slightly rewording the question.
  6. Coherence Metrics: They developed a rule-based matcher to extract the "implied answer" from the natural language explanation to see if it matched the selected multiple-choice label.
The proposed evaluation methodology showing the original condition and five perturbations designed to test grounding and robustness.

Datasets

The study utilized two major mammography benchmarks from the MammoVQA suite:

  • RSNA Screening Mammography: Focused on BI-RADS assessment and breast density (ACR categories).
  • EMBED: A large-scale, racially diverse dataset used for exam-level BI-RADS categorization.

Key Insights and Ideas

  • Confidence-Evidence Inversion: Ideally, a model’s confidence should drop when the image is removed. Instead, models like MedGemma remained highly confident (~95%) in a specific category even when looking at a blank gray square.
  • The "None of the Above" (NOTA) Failure: When images were replaced with junk data, a grounded model should select NOTA. Most models failed this; MedGemma, for instance, selected a clinical category 100% of the time for unrelated photos, showing a total lack of "epistemic humility."
  • Option-Order Fragility: Simply shuffling the order of A, B, C, and D flipped the models' answers in 40% to 99% of cases. This suggests the models are "multiple-choice selectors" rather than "image interpreters."
  • Self-Contradictory Rationale: The authors found cases where a model's reasoning was clinically sound but its final answer was wrong, or vice versa. Accuracy metrics would credit a model for a correct label even if the reasoning proved it didn't understand the image.

Key Results

The quantitative results paint a sobering picture of current VLM capabilities in specialized medical tasks:

  • Low Baseline Accuracy: On BI-RADS tasks, no model exceeded the majority-class prior (the accuracy you'd get by just guessing the most common label).
  • High Flip Rates: Under the "Perturbed-Options" condition, MedGemma's accuracy dropped by 24.8 percentage points on BI-RADS just by reordering the choices.
  • Reasoning Disconnect: In the EMBED dataset, Lingshu-7B's explanation named the correct category 57.6% of the time, yet it only selected the correct multiple-choice answer 10.3% of the time.
Table showing output accuracy and "None of the Above" selection rates across different models and datasets.

Limitations and Future Work

The authors acknowledge several limitations:

  • Format Constraints: The study is limited to multiple-choice questions; free-text report generation might yield different coherence patterns.
  • Model Scope: Only open-weights models in the 4-8B parameter range were tested. The behavior of larger, closed models (like GPT-4o) remains unknown in this specific mammography context.
  • Clinical Realism: The perturbations did not include clinically nuanced changes like swapping laterality (left vs. right breast) or comparing against prior images, which are standard in radiologist workflows.

Future work should explore whether domain-specific fine-tuning can actually fix these coherence failures or if they are fundamental to the architecture of current vision-language transformers.

Summary updated

Abstract

As vision-language models (VLMs) move toward interpreting medical images directly, their outputs increasingly include not just a label but a stated confidence and a written justification. Yet the field still predominantly evaluates VLMs using just accuracy. We evaluate the coherence between answer, confidence, and explanation across four VLMs on breast density and BI-RADS categorization, using two mammography datasets. We perform controlled perturbations of the question, multiple choice option orders, and the images and assess changes in the three outputs. Our results suggest confidence does not decrease even as the mammography image is changed. Shuffling the options also changes most VLMs' answers. Notably, there are cases where a model's own stated reasoning contradicts its final answer, a distinction accuracy alone cannot capture but one that governs whether an AI-generated answer can be trusted.
BibTeX
@misc{nam2026evaluating,
  author = {Rebeca Nam and Chase M. Fensore and Christian A. Fauria-Robinson and Judy Gichoya and Joyce C. Ho},
  title = {{Evaluating Cross-Signal Coherence in Mammography Vision-Language Models}},
  year = {2026},
  eprint = {2611.12574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12574},
}
Figure