Vision-language models are models that read an image and answer questions about it in text. They are starting to be pointed at pathology slides, but they have a known habit: when the picture and their memorized knowledge disagree, they often side with what they have read. In tumor immunology, what they have read is very consistent. The literature says that tumors in cancers like pancreatic cancer are surrounded by suppressive macrophages, the immune cells that "eat" other cells. That same literature is why therapies that block the CD47 "don't eat me" signal were developed. We propose to measure, tier by tier, when a model stops reading a multiplex-stained tissue image and starts reciting that story. The test uses real tiles whose ground truth contradicts the textbook, for example tumor nests with almost no macrophages nearby.
The central claim is that textbook bias in tissue images is not uniform. It should be small when the model is asked to find and count cells, larger when it is asked about spatial relationships, and largest when it is asked whether a tumor region offers a macrophage population that an anti-CD47 therapy could act on. The most revealing symptom should be internal inconsistency: a model counts few macrophages near a tumor nest, then concludes that the nest is macrophage-rich. We will build this three-tier diagnostic from existing segmented multiplex data. We will then test one inference-time correction, contrasting the model's answer on the full tile against its answer when the tissue within 100 µm of the tumor nest is masked. Finally, we will check whether that correction beats the simpler fixes a skeptic would reach for first.
Core Claim
Prior work has shown textbook bias in medical images, but only at the level of gross anatomy and a single task type. Your other Left! found that models judging relative organ positions on abdominal CT fall to chance when images are flipped. It also found that GPT-4o rises to 87.4% when organ names are replaced with abstract letters, which shows the medical vocabulary itself triggers the bias. That paper motivates our work. Our question is different: does the bias grow as the question climbs from perception toward mechanism, and does the model's high-level answer contradict its own low-level readings?
Tiered hallucination diagnostics already exist for everyday photographs:
- PhD splits object, attribute, position and counting tasks across misleading-context and counter-common-sense modes.
- HallusionBench uses edited control images and a decision tree to separate language hallucination from visual illusion.
Both stay on natural images and charts, and neither asks whether answers at different tiers agree with each other on the same image. Our deliverable is therefore a diagnostic built around one quantity, the tier-consistency gap. We also contribute a tested correction, but the diagnostic comes first.
Research Program
Data. The substrate is the kind of data described in A Novel Framework for Characterization of Tumor-Immune Spatial Relationships in Tumor Microenvironment. That work covers 23 multiplex immunohistochemistry whole-slide images of pancreatic cancer. Multiplex immunohistochemistry stains several cell-type markers on one slide, here CD4, CD8, CD16, CD163 for tumor-associated macrophages, pancytokeratin and K17. A U-Net ensemble, a segmentation network, detects and classifies cells in these slides, with F1 up to 0.99 for K17-negative cells. The framework also defines a 100 µm "influence area" around each tumor nest.
This paper supplies our ground truth, not a competitor. Its cell calls and nest masks let us label thousands of tiles automatically. Its finding that CD163 macrophages are distributed far from randomly (mean absolute deviation 0.99) means the tiles vary a lot. That variation gives us natural counter-textbook tiles without any image editing. We do not know from the file whether this cohort is public. If it is not, we will reproduce the pipeline on a comparable multiplex cohort with the same markers.
A caveat shapes the mechanism tier. None of these markers is CD47. That tier therefore asks about the therapy's substrate rather than the checkpoint itself. The reviews Targeting the CD47/SIRPα pathway in malignancies and Targeting tumor‑associated macrophages say that blocking CD47 works only if macrophages are present and engaged at the tumor. The question is whether such a population exists adjacent to the nest. That makes the tier-3 answer derivable from tier-1 and tier-2 facts, which is exactly what makes consistency checkable.
The three tiers. Each tile is asked a set of questions at three levels:
- Tier 1 (perception): how many CD163-positive cells are in the marked region.
- Tier 2 (relation): whether most macrophages lie inside or outside the nest's 100 µm influence area.
- Tier 3 (mechanism): whether this nest has a peritumoral macrophage population that CD47 blockade could plausibly engage, and whether the environment looks macrophage-dominated or T-cell-dominated.
The tier-3 ground truth is a fixed rule applied to the segmentation-derived tier-1 and tier-2 values, so no pathologist judgment is needed.
Three controls against the textbook.
- Counter-prior tiles. Real tiles are stratified by whether they agree with the canonical story. "Prior-conforming" tiles are nests ringed by CD163 cells. "Counter-prior" tiles have sparse peritumoral macrophages, or CD8 T cells dominating the influence area.
- Legend swaps. The tile is rendered with pseudo-color channels and a text legend. The legend is then permuted, so that the channel called "CD163" really shows CD8 cells, and the ground truth follows the legend. A model that looks at the image and the legend must change its answer. A model reciting the textbook will not.
- Name-blinding. Following Your other Left!, marker names are replaced with letters. This is possible at tiers 1 and 2 but not honestly at tier 3, because the tier-3 question is about the named biology.
Metrics. We report three:
- Prior-conformity rate: on counter-prior tiles, the fraction of answers that match the textbook rather than the image.
- Tier-consistency gap: the fraction of tiles where the model's tier-3 answer contradicts the rule applied to its own tier-1 and tier-2 answers.
- Plain accuracy per tier.
The correction. The correction is a contrastive decoder, meaning one that compares the model's word probabilities under two versions of the input. Let $\ell(y \mid x, q)$ be the model's output logits for answer token $y$, given tile $x$ and question $q$. Let $x_{\setminus N}$ be the same tile with the nest's 100 µm influence area filled with background. We decode from
$$ p(y) \propto \exp\big((1+\alpha)\,\ell(y \mid x, q) - \alpha\,\ell(y \mid x_{\setminus N}, q)\big), $$
where $\alpha \ge 0$ sets how strongly we subtract what the model would say without seeing the niche. This is the functional form used by Visual Contrastive Decoding, which contrasts against a Gaussian-noised image. It is also the form of Contrastive Region Guidance, which contrasts against a blacked-out bounding box. Our only change is the mask: an irregular, biologically defined region taken from the segmentation. We present this honestly as a CRG variant. Its value, if any, lies in whether the biological shape of the mask matters, and the ablations below test that directly.
Layer-wise analysis. For open models we add an early-exit analysis. Towards Understanding How Knowledge Evolves in Large Vision-Language Models showed that in LLaVA-1.5, grounded predictions formed around layer 25 "mutate" into high-frequency words by layer 32. We ask whether correct tier-3 answers on counter-prior tiles appear in middle layers and are overwritten late. We also ask whether that overwrite is more pronounced at tier 3 than at tier 1.
Models. We evaluate:
- LLaVA-1.5 at 7B and 13B. This pair lets us test the "size paradox" of PhD, where the larger backbone grounds worse on base images.
- LLaVA-1.6-34B, the model on which CRG reported its largest gain.
- GPT-4o through its API, for tiers only, since logits are unavailable.
- QuiltNet and CONCH, the pathology image-text models adapted in Effortless Vision-Language Model Specialization in Histopathology without Annotation, scored as zero-shot classifiers at tiers 1 and 2. That paper's adaptation on uncurated captions boosted SICAP weighted kappa from 0.02 to 0.50. We will test whether the same adaptation raises prior-conformity on counter-prior tiles.
Baselines
Plain greedy decoding. The do-nothing reference is plain greedy decoding of each model on each tier. Every other method must be compared against it on the identical tile set and prompts. There is no reported result found on Lacuna for any vision-language model on multiplex tumor-immune tiles, so these reference numbers will be produced here.
VCD. The first strongest existing method is Visual Contrastive Decoding. It is the standard training-free way to subtract a language prior. The paper reports up to +7.4 F1 on POPE's MSCOCO random split and an MME total rising from 565 to 604 for LLaVA-1.5.
CRG with bounding boxes. The second is Contrastive Region Guidance with rectangular masks: the tight box around the influence area. CRG reports +11.1% absolute accuracy for LLaVA-1.6-34B on ViP-Bench and +10% on What'sUp.
Fairness conditions. Both contrastive methods get the same two forward passes per token as ours. Each gets the same grid search over $\alpha$ (five values), tuned on a held-out split of prior-conforming tiles only. This keeps the counter-prior test set untouched by any method. Beating them means a lower prior-conformity rate on counter-prior tiles and a smaller tier-3 consistency gap, with tier-1 accuracy no worse, across bootstrap resamples at the slide level. Resampling slides rather than tiles prevents tiles from the same slide from inflating significance.
Mask-specificity ablation. This ablation isolates the key ingredient. It replaces the niche mask with a random irregular mask of equal area placed away from the nest, and separately with a mask of the tumor nest itself. If the random mask does as well, the benefit is generic image corruption, as in VCD, rather than removing the region the question depends on.
Segment-then-reason pipeline. The final baseline feeds the U-Net's cell counts and distances as text to the same language model and asks the tier-3 question. It has access to exactly the ground-truth features, so it is a hard target.
Why not just…?
Why not just replace the marker names with letters, as the CT study did? Name-blinding produced the largest effect reported in the file, GPT-4o reaching 87.4% on abstract markers in Your other Left!. We expect it to work well at tiers 1 and 2 here too, and we run it as a baseline there. It cannot be applied at tier 3. Asking whether macrophages are available for CD47 blockade without saying "macrophage" or "CD47" is a different question. The tier-consistency gap exists precisely to show whether the bias concentrates where this fix is unavailable. If the gap is near zero at tier 3, name-blinding suffices and our correction is unnecessary. We would report that outcome as the result.
Why not just use VCD or CRG unchanged? They may be enough, and they are our main baselines, not straw men. The reasons for doubt are specific. Gaussian noise corrupts nuclear and membrane staining everywhere at once, so it cannot tell the model which region the answer should come from. Boxes around organically shaped nests include large amounts of tumor and distant stroma. The mask-specificity ablation settles this. If box-masked CRG matches the niche mask, the contribution reduces to the diagnostic benchmark, which we would still consider worth publishing.
Why not just count cells with the U-Net and let a text model reason from the numbers? This is the most serious threat. The U-Net's near-perfect cell calls in A Novel Framework for Characterization of Tumor-Immune Spatial Relationships in Tumor Microenvironment could make the vision-language model redundant for these questions. We run this pipeline as a baseline and also subject it to the counter-prior test. A text model handed "few CD163 cells within 100 µm" may still declare the region macrophage-dominated from its reading. If it does not, the practical conclusion is that perception should be delegated to a specialist and the image-reading model avoided at tier 3. We would state that plainly.
Why not just fine-tune or use a bigger model? Scale does not reliably fix this in the file's evidence. PhD found 13B backbones ground worse than 7B on base images. HallusionBench reports GPT-4V at 31.42% question-pair accuracy. CRG still helped the prompt-fine-tuned ViP-LLaVA. Our 7B/13B/34B sweep, together with the before-and-after comparison of pathology adaptation on QuiltNet and CONCH, tests this directly rather than assuming it.
Decisive Checks
The existence claim. The claim that textbook bias rises with tier stands or falls on two numbers from unmodified models. First, prior-conformity on counter-prior tiles must be clearly higher at tier 3 than at tier 1. Second, the tier-consistency gap must be substantially above what we see on prior-conforming tiles. If models are equally wrong at every tier, the failure is perceptual, not linguistic, and the hierarchy adds nothing. The legend-swap control separates these two explanations: a perceptual failure should not depend on what the legend says.
The correction claim. The correction must lower counter-prior conformity at tier 3 relative to box-masked CRG and VCD at matched cost, and it must beat the random-irregular-mask ablation.
Transfer. We rerun the same masked contrast on MIRP CT slices. There, TotalSegmentator organ masks provide the regions, and name-blinding's 87.4% gives an external reference point for how much of the textbook effect an inference-time method recovers.
The layer analysis. This analysis is judged by a single prediction. On counter-prior tiles, the correct tier-3 answer should be more probable at the middle critical layer than at the output. Only if that holds is it fair to say the image evidence was present and then overwritten.
References
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- A Novel Framework for Characterization of Tumor-Immune Spatial Relationships in Tumor Microenvironment
- Targeting the CD47/SIRPα pathway in malignancies: recent progress, difficulties and future perspectives
- Targeting tumor‑associated macrophages: Critical players in tumor progression and therapeutic strategies (Review)
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Contrastive Region Guidance: Improving Grounding in Vision-Language Models Without Training
- Towards Understanding How Knowledge Evolves in Large Vision-Language Models
- Effortless Vision-Language Model Specialization in Histopathology without Annotation