When the Report Contradicts the Slide: Discordance Testing for Pathology Vision-Language Models

Pathology AI systems now read a tissue image together with text: a caption, an immunostain panel, a clinical note. A safe system should notice when the two disagree. A real failure case looks like this. The image shows normal colon lining, but the attached text describes a carcinoma that belongs to a different patient. The text-only benchmark Pathology's Last Exam found that large language models rarely refuse in such cases. Instead they invent a confident diagnosis, and even the best model declined only half of the specimen mix-ups. Nobody has asked the same question with real pixels in the loop. This proposal builds that test. It pairs real histology images with clinical text that is internally consistent but contradicts the image. It then measures whether pathology vision-language models flag the conflict or simply adopt the text's diagnosis.

There is a twist. A model that does catch a mismatch may be catching the wrong thing. Histology datasets carry lab-specific stain colors and compression signatures, and pathology image-text models are trained on figures scraped from journal articles. A model could therefore "detect" a mix-up because the image and the text come from different laboratories, not because the tissue contradicts the diagnosis. We use co-authorship mapping, which draws the network of who publishes with whom, to recover lab-of-origin groups where no site label exists. That lets us build swaps within a research community and across communities. The decisive question is whether discordance detection survives once provenance cues are removed.

Core Claim

We make two claims. The first is that pathology vision-language models defer to contradicting text far more often than they defer to the image. This is a cross-modal version of the "sycophancy trap", the tendency to agree rather than refuse. The second is that whatever mismatch detection these models show is inflated when image and text come from different collaboration communities. Within-community swaps with the same degree of diagnostic conflict should be detected markedly less often. If both hold, current evaluations overstate cross-modal safety. They also suggest a concrete fix: evaluation splits and detection training should be grouped by provenance.

How This Relates to Prior Work

Three papers motivate the gap. Pathology's Last Exam built 20 adversarial text cases: lineage clashes, meaning marker combinations that cannot coexist biologically, and patient mix-ups. It showed that models overclaim rather than use an insufficiency option. Its cases are entirely text, so any contradiction there can in principle be caught by reading alone. How Good is my Histopathology Vision-Language Foundation Model? introduced Histo-VL, which spans 26 organs, 14 cohorts and up to 11 scanners. It showed that CONCH swings up to 26% in balanced accuracy under caption rewording and that its calibration worsens under prompt ensembling. Histo-VL perturbs captions and pixels but never makes them contradict each other. In general vision-language safety, Safe Inputs but Unsafe Output and Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios show that individually benign inputs can combine into an unsafe output. Our cases are the clinical analogue: each input is plausible alone, and only the combination is wrong.

One paper constrains the design. NCT-CRC-HE: Not All Histopathological Datasets are Equally Useful showed that color histograms alone reach 82.2% accuracy on a standard colorectal benchmark because class labels correlate with acquisition artifacts. Any image-text agreement score may therefore be partly an agreement on provenance. Our within-community controls exist because of this result.

Two papers supply the method. A visual-language foundation model for computational pathology (CONCH) learns image-text alignment from 1.17M PubMed Central and educational figure-caption pairs. We reuse its image-text similarity as a mismatch score and its figure-extraction pipeline to build a test corpus with author metadata. Applying centrality measures to impact analysis and CollaborationViz supply the co-authorship graph construction and centrality tools.

Ecosystem Graphs treats the foundation-model supply chain as a dependency graph but stops at organizations and general-purpose models. We go down to author-level communities in a biomedical corpus. We do not study deliberately planted backdoors of the kind in Architectural Backdoors in Vision-Language Model Supply Chains. Our concern is naturally occurring shortcuts, which that work leaves open.

Research Program

The first component is a discordance test set. It is built from images with trusted labels: Histo-VL cohorts, the CRC-VAL-HE-7K split of NCT-CRC-HE, and TCGA breast cancer slides. We attach three kinds of text.

The first is concordant text, a short report consistent with the image label.

The second is image-discordant text. This is a report that is internally coherent, so a text-only reader sees nothing wrong, but whose diagnosis and supporting stains contradict the image's ground truth. An example is a normal mucosa tile paired with a coherent adenocarcinoma workup.

The third is the mix-up: the genuine report of another case, sampled at controlled diagnostic distance from the true label.

We deliberately exclude text-internal lineage clashes from the core test, because a text model can solve those without the image. They are kept only as a reference condition. A board-certified pathologist reviews a sample of the generated reports for coherence.

The second component is a provenance-annotated corpus. Following CONCH's extraction approach, we collect open-access PubMed Central histopathology figures with their captions and author lists. From the author lists we build a co-authorship graph and partition it into collaboration communities. Each figure inherits the community of its authors. On TCGA we validate this proxy against tissue-source-site codes: if co-authorship communities do not agree with site groupings better than chance, the proxy is not trusted. Mix-ups are then constructed in two matched versions. In the within-community version, image and text come from the same community. In the cross-community version, they come from different communities. Both versions are matched on organ and diagnostic distance.

The third component is the evaluation protocol, which has two settings.

The first setting covers contrastive models such as CONCH, scored on image-text similarity. We measure how well that similarity separates concordant from discordant pairs.

The second setting covers generative vision-language models: PathAlign (PathAlign), plus general models that accept images, among those evaluated in Pathology's Last Exam. These are given an explicit "inconsistent / insufficient" answer option, following that paper's protocol.

We record four outcomes. The flag rate is how often the model refuses or flags. The text-capitulation rate is the fraction of discordant cases where the output diagnosis matches the text rather than the image. The image-faithful rate is how often the diagnosis follows the image. We also measure calibration error on the discordant cases.

Baselines

The naive reference point is a system that never flags and always outputs a diagnosis. Its capitulation rate equals whatever the model does by default. A random flagger matched to the model's overall flag rate gives the chance line for detection.

The strongest existing reference on this exact failure is the refusal behavior reported in Pathology's Last Exam: GPT-5 refused 70% of adversarial cases and 50% of mix-up cases, on text only. We run the same generative models under the same insufficiency-option prompt with the image added. This tells us whether the image helps or hurts refusal relative to that text-only figure. A model that clears 50% on our image-based mix-ups would beat the published mix-up refusal rate. It would only count as real progress if it also holds on within-community mix-ups.

The strongest method-level competitor is CONCH's own image-text similarity used as a zero-shot mismatch detector. It is the most direct application of the best-documented pathology alignment model, which reports 44.0% mean retrieval recall versus 17.3% for the next model. Retrieval recall is not the same as mismatch detection, and no reported result was found on Lacuna for CONCH as a discordance detector. We therefore report its detection AUROC from scratch. Its threshold is tuned on a held-out set of communities, and all detectors get the same tuning budget.

A text-only consistency checker is a large language model that sees only the report. It establishes that our discordant texts are not detectable without the image. If this checker exceeds chance on the core set, the affected cases are regenerated.

A provenance-only probe is a color-histogram classifier, the 82.2% shortcut from the NCT-CRC-HE study, trained to predict collaboration community from the image. It quantifies how much lab identity is recoverable from pixels alone.

The ablation that isolates our key ingredient replaces community-grouped mix-ups with ordinary random patient-level mix-ups. If detection numbers do not differ between the two, the provenance lens adds nothing.

Why not just…?

Why not just use tissue-source-site labels instead of building a co-authorship graph? Where site labels exist, as in TCGA, they are better, and we use them to validate the graph. This is a real threat on TCGA. However, the PubMed Central figures that pathology vision-language models are trained on carry no site label, and author lists are the only provenance signal attached to them. The proposal settles the question empirically. If co-authorship communities fail to reproduce the site-code results on TCGA, we drop the graph and restrict the study to labeled cohorts.

Why not just stain-normalize the images to remove the shortcut? Normalization targets color, but the NCT-CRC-HE study shows shortcuts also live in dynamic range and JPEG compression. Caption phrasing styles shared within a lab are untouched by any image normalization. We include stain normalization as an extra condition. If it closes the within- versus cross-community detection gap by itself, that is a useful and simpler result, and we would report it as the main finding.

Why not just reuse the text-only benchmark with an image appended? Pathology's Last Exam's lineage clashes are solvable from text alone. Appending an image would measure whether the image distracts the model, which is the effect Images are Achilles' Heel of Alignment reports even for blank images. It would not measure whether the model uses the image to catch a conflict. We run this condition as a secondary check of the distraction effect in pathology. Only image-discordant cases with coherent text test the cross-modal gap itself.

Why not just threshold CONCH similarity and call it a safety filter? That is our strongest baseline, not a dismissal. The worry is that its scores reward provenance agreement. If its AUROC is equal on within- and cross-community swaps, the simple filter is adequate, and we would say so.

Decisive Checks

The first check decides whether the cross-modal sycophancy claim holds. It requires that generative models' text-capitulation rate on image-discordant cases exceed their image-faithful rate. A negative result here is also publishable: it would mean images protect better than text alone does.

The second check decides the provenance claim. It requires a material drop in detection, by AUROC for CONCH and by flag rate for the generative models, from cross-community to within-community mix-ups at matched diagnostic distance. Confidence intervals come from bootstrapping over communities rather than images, because cases within a community are not independent. The claim also requires that the provenance probe predicts community from pixels well above chance; this ties any gap to recoverable lab signatures.

Calibration error on discordant cases, compared with the 0.53 to 0.61 CONCH range reported by Histo-VL on ordinary inputs, shows whether models are overconfident specifically when the evidence conflicts.

If both main effects hold, we release the community-grouped splits as a standard evaluation. Any future mismatch detector would then have to be tested where lab identity cannot help it.