TL;DR
This paper exposes "extent leakage" in text-guided medical image segmentation benchmarks, where captions contain size and location information derived directly from the ground-truth masks. The authors demonstrate that current models primarily "read" the answer from these prompts rather than learning from the images, leading to inflated performance metrics and a decreased ability to segment from images alone.
What Problem Does This Paper Address?
In the burgeoning field of vision-language medical AI, text-guided segmentation models are designed to use short captions (e.g., "one small polyp") to improve accuracy. However, a critical flaw exists in how these models are evaluated: the captions in popular benchmarks are often generated using the very ground-truth masks the models are supposed to predict.
In a real clinical setting, a doctor wouldn't provide a prompt like "a polyp covering exactly 11% of the image" because the mask doesn't exist yet. This creates a "Clever Hans" effect—named after a horse that appeared to do math but was actually reading subtle cues from its trainer. The authors argue that this label leakage (specifically extent leakage) overestimates model performance and masks a fundamental failure: models are learning to rely on "cheating" text cues instead of the actual medical imagery.
Methods
The researchers employed a "counterfactual" approach to dissect model behavior:
- Caption Edits: They systematically edited test-time captions—removing size words, using mismatched captions from other images, or forcing a "small" lesion to be called "large"—to see how much the model's prediction relied on specific words.
- Retraining Study: They retrained models on "size-blind" or "text-free" versions of the datasets to see if the dependence on leaked information could be broken and how it affected image-only performance.
- Caption-only Leakage Score ($R^2_{cap}$): They developed a model-free metric that uses simple regression to see how well the lesion's area can be predicted from the caption text alone. If the text alone can predict the area, the benchmark is "leaking" the label.
- ExtentSeg: They introduced a new architecture that estimates lesion extent (size/location) directly from the image to guide segmentation, providing a more realistic "annotation-free" baseline.
Datasets
The study audited seven captioned test sets across three modalities:
- Colonoscopy: Kvasir-SEG (training), CVC-ClinicDB, CVC-300, CVC-ColonDB, and BKAI-IGH.
- Breast Ultrasound: BUSI.
- Chest X-ray: QaTa-COV19.
The authors noted that for colonoscopy and ultrasound, captions were generated via a fixed template based on mask area thresholds. For X-rays, captions were written by annotators but still heavily focused on the lesion's extent (lung zones).
Key Insights and Ideas
- Extent is Everything: On colonoscopy and X-ray datasets, the "extent attribute" (size word or lung-zone phrase) accounts for nearly all the performance gain attributed to text. Other descriptive words like color or shape contribute almost nothing.
- The "Lazy" Model Habit: Models trained with leaked captions become "lazy." When tested without text, they perform significantly worse than models that were never given the leaked text during training. The leakage effectively "weakens" the model's ability to learn from the image itself.
- Leakage is Quantifiable: The $R^2_{cap}$ score showed that caption words alone could predict lesion area with a coefficient of determination between 0.54 and 0.77 on most test sets. This is a clear signal of label leakage.
- A Polyp is a Rock: Models are so focused on the extent words that they ignore the actual nouns. Calling a polyp a "rock" in the prompt didn't reduce segmentation performance, as long as the size word remained accurate.
Key Results
- Gain Recovery: Keeping only the size/extent word in a caption recovered 84% to 98% of the performance improvement over an empty caption.
- Performance Loss: Removing just the extent attribute caused models to lose 52% to 77% of their text-driven gains.
- Image-Only Performance: Models trained without text (Text-free) outperformed the original models by 2.39 to 11.28 Dice points when both were tested on images alone.
- ExtentSeg Success: The proposed
ExtentSegmodel, which estimates extent from the image rather than reading it from a mask-derived caption, reached 74.59 Dice on colonoscopy—approaching the published "leaked" result of 77.23 without using any information from the test mask.
Limitations and Future Work
The authors acknowledge that their study focused on specific architectures (MedCLIPSeg and RecLMIS) and that ExtentSeg was only evaluated on colonoscopy. Furthermore, the study does not test real-world, clinician-written descriptions, which are likely to be far more subjective and uncertain than the rigid templates used in current benchmarks.
Future work should focus on developing benchmarks with "clinically plausible" prompts and adopting the authors' proposed evaluation protocol, which includes reporting results with empty and mismatched captions to isolate true image-text interaction from simple label leakage.
Summary updated