Counterfactual Analysis of Extent Leakage from Annotation-Derived Prompts in Vision-Language Medical Image Segmentation

Niran Nataraj, Ankita Das, C. Krishna Mohan, Kenji KawashimaarXiv 2026 · arXiv

TL;DR

This paper exposes "extent leakage" in text-guided medical image segmentation benchmarks, where captions contain size and location information derived directly from the ground-truth masks. The authors demonstrate that current models primarily "read" the answer from these prompts rather than learning from the images, leading to inflated performance metrics and a decreased ability to segment from images alone.

What Problem Does This Paper Address?

In the burgeoning field of vision-language medical AI, text-guided segmentation models are designed to use short captions (e.g., "one small polyp") to improve accuracy. However, a critical flaw exists in how these models are evaluated: the captions in popular benchmarks are often generated using the very ground-truth masks the models are supposed to predict.

In a real clinical setting, a doctor wouldn't provide a prompt like "a polyp covering exactly 11% of the image" because the mask doesn't exist yet. This creates a "Clever Hans" effect—named after a horse that appeared to do math but was actually reading subtle cues from its trainer. The authors argue that this label leakage (specifically extent leakage) overestimates model performance and masks a fundamental failure: models are learning to rely on "cheating" text cues instead of the actual medical imagery.

Figure 1: A published text-guided segmenter reads the lesion’s extent from its caption. Each image is shown with the empty caption and with its own caption in which only the extent has been changed. Top: the size word of a colonoscopy caption, which is computed from the mask. Bottom: the lung-zone phrase of a chest X-ray caption, which was written by annotators. Numbers are Dice scores.

Methods

The researchers employed a "counterfactual" approach to dissect model behavior:

  • Caption Edits: They systematically edited test-time captions—removing size words, using mismatched captions from other images, or forcing a "small" lesion to be called "large"—to see how much the model's prediction relied on specific words.
  • Retraining Study: They retrained models on "size-blind" or "text-free" versions of the datasets to see if the dependence on leaked information could be broken and how it affected image-only performance.
  • Caption-only Leakage Score ($R^2_{cap}$): They developed a model-free metric that uses simple regression to see how well the lesion's area can be predicted from the caption text alone. If the text alone can predict the area, the benchmark is "leaking" the label.
  • ExtentSeg: They introduced a new architecture that estimates lesion extent (size/location) directly from the image to guide segmentation, providing a more realistic "annotation-free" baseline.

Datasets

The study audited seven captioned test sets across three modalities:

  • Colonoscopy: Kvasir-SEG (training), CVC-ClinicDB, CVC-300, CVC-ColonDB, and BKAI-IGH.
  • Breast Ultrasound: BUSI.
  • Chest X-ray: QaTa-COV19.

The authors noted that for colonoscopy and ultrasound, captions were generated via a fixed template based on mask area thresholds. For X-rays, captions were written by annotators but still heavily focused on the lesion's extent (lung zones).

Figure 2: How benchmark captions are derived from ground-truth masks. The size word is a direct function of the mask area.

Key Insights and Ideas

  • Extent is Everything: On colonoscopy and X-ray datasets, the "extent attribute" (size word or lung-zone phrase) accounts for nearly all the performance gain attributed to text. Other descriptive words like color or shape contribute almost nothing.
  • The "Lazy" Model Habit: Models trained with leaked captions become "lazy." When tested without text, they perform significantly worse than models that were never given the leaked text during training. The leakage effectively "weakens" the model's ability to learn from the image itself.
  • Leakage is Quantifiable: The $R^2_{cap}$ score showed that caption words alone could predict lesion area with a coefficient of determination between 0.54 and 0.77 on most test sets. This is a clear signal of label leakage.
  • A Polyp is a Rock: Models are so focused on the extent words that they ignore the actual nouns. Calling a polyp a "rock" in the prompt didn't reduce segmentation performance, as long as the size word remained accurate.

Key Results

  • Gain Recovery: Keeping only the size/extent word in a caption recovered 84% to 98% of the performance improvement over an empty caption.
  • Performance Loss: Removing just the extent attribute caused models to lose 52% to 77% of their text-driven gains.
  • Image-Only Performance: Models trained without text (Text-free) outperformed the original models by 2.39 to 11.28 Dice points when both were tested on images alone.
  • ExtentSeg Success: The proposed ExtentSeg model, which estimates extent from the image rather than reading it from a mask-derived caption, reached 74.59 Dice on colonoscopy—approaching the published "leaked" result of 77.23 without using any information from the test mask.
Figure 3: Examples of counterfactual edits used to test model sensitivity to specific caption attributes.

Limitations and Future Work

The authors acknowledge that their study focused on specific architectures (MedCLIPSeg and RecLMIS) and that ExtentSeg was only evaluated on colonoscopy. Furthermore, the study does not test real-world, clinician-written descriptions, which are likely to be far more subjective and uncertain than the rigid templates used in current benchmarks.

Future work should focus on developing benchmarks with "clinically plausible" prompts and adopting the authors' proposed evaluation protocol, which includes reporting results with empty and mismatched captions to isolate true image-text interaction from simple label leakage.

Summary updated

Abstract

Text-guided segmentation models receive an image and a short caption about the lesion, and on public benchmarks the caption improves accuracy considerably. On colonoscopy and breast ultrasound, however, the size, count and location words of each caption were computed from the mask that later scores the prediction, and the chest X-ray captions were written by benchmark annotators. We tested what two published models take from them. On colonoscopy and chest X-ray, it is mostly the lesion's extent. Keeping only the size word or the lung-zone phrase recovers 84 to 98 percent of the improvement over an empty caption, and removing it loses 52 to 77 percent. The rest matters little, and a polyp called a rock is segmented just as well. We call this extent leakage. On breast ultrasound, nearly half the captions also name the diagnosis, which carries part of the benefit. Retraining on three datasets shows that models pick up this habit from their training captions, though only partly on breast ultrasound, and that it weakens their segmentation from the image alone. The leak can be measured without training a model, since caption words alone predict lesion area (coefficient of determination 0.54 to 0.77 on six of seven test sets). Without the test mask, the best method we tested, which estimates extent from the image, reaches 74.59 Dice on colonoscopy against the published 77.23. We propose an evaluation protocol that keeps what a caption says about the image apart from what it copies from the label.
BibTeX
@misc{nataraj2026counterfactual,
  author = {Niran Nataraj and Ankita Das and C. Krishna Mohan and Kenji Kawashima},
  title = {{Counterfactual Analysis of Extent Leakage from Annotation-Derived Prompts in Vision-Language Medical Image Segmentation}},
  year = {2026},
  eprint = {2611.13970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13970},
}
Figure