MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile, Yiming Xiao, Hassan RivazarXiv 2026 · arXiv

TL;DR

MedCLIPSeg introduces a probabilistic, bidirectional vision-language adapter framework that tailors CLIP for medical image segmentation. By modeling visual and textual features as probability distributions rather than fixed vectors, it achieves exceptional data efficiency and robustness against domain shifts while generating well-calibrated, pixel-level uncertainty maps for clinical decision support.

What Problem Does This Paper Address?

Deep learning has revolutionized medical image segmentation, but standard architectures (like U-Net variants) face three major hurdles in clinical deployment:

  1. Annotation Scarcity: Expert pixel-level masks are incredibly expensive and time-consuming to produce.
  2. Anatomical Ambiguity: Lesion boundaries are often fuzzy due to gradual intensity transitions or artifacts, making deterministic boundaries inherently flawed.
  3. Domain Shift: A model trained on data from one hospital or scanner often fails dramatically when deployed on another.

Worse yet, conventional deterministic models are systematically overconfident—they will confidently output an incorrect segmentation mask without warning the clinician. While Vision-Language Models (VLMs) like CLIP offer strong cross-modal representations that could help, adapting them for dense, pixel-level medical prediction without losing their generalization capabilities remains an open challenge.

Methods

MedCLIPSeg addresses these challenges by introducing a novel Probabilistic Vision-Language (PVL) Adapter that connects CLIP’s frozen image and text encoders across multiple deep layers.

Overview of the MedCLIPSeg framework showing the integration of PVL Adapters and uncertainty map generation.

The core technical innovations include:

  • Probabilistic Attention ($\text{Attn}_{\text{PVL}}$): Instead of projecting inputs into deterministic Key ($K$) and Value ($V$) vectors, the PVL adapter models them as Gaussian distributions with learnable means and variances. This allows the model to explicitly capture aleatoric uncertainty (from noisy data) and epistemic uncertainty (from unseen domains).
  • Confidence-Weighted Attention: The attention mechanism subtracts a variance-based penalty from the standard dot-product similarity score. This naturally downweights uncertain tokens, preventing noisy or ambiguous features from corrupting the multi-modal alignment.
  • Bidirectional Interaction: Visual tokens update textual tokens and vice versa through a two-way Transformer layer, enabling deeper contextual fusion than traditional unidirectional text-to-vision prompting.
  • Soft Patch-Level Contrastive Loss ($L_{\text{SoftCon}}$): To improve data efficiency, the authors align average-pooled visual patch embeddings with text prompts. Because medical text prompts within a batch can be highly similar, they replace hard contrastive targets with soft targets derived from text-to-text similarities.

At test time, Monte Carlo (MC) sampling is performed over multiple stochastic forward passes. The mean of these samples forms the final segmentation mask, while the predictive entropy yields a pixel-level uncertainty map.

Figure 3. Illustrations of PVL Adapter and AttnPVL.

Datasets

The authors evaluated MedCLIPSeg across 16 datasets spanning 5 imaging modalities (Ultrasound, MRI, Dermatoscopy, Endoscopy, X-ray) and 6 organs:

  • In-Distribution (ID) / Data Efficiency: BUSI (breast), BTMRI (brain), ISIC (skin), Kvasir-SEG (colon), QaTa-COV19 (chest), and EUS (pancreas).
  • Out-of-Distribution (OOD) / Generalization: BUSUC, BUSBRA, BUID, UDIAT (breast ultrasound); BRISC (brain MRI); UWaterlooSkinCancer (skin); CVC-ColonDB, CVC-ClinicDB, CVC-300, and BKAI (polyp endoscopy).

For datasets lacking text descriptions, the authors introduced an automated pipeline using GPT-5 templates filled with lightweight image/mask attribute extractions (e.g., location, shape, color).

Key Insights and Ideas

  • Embracing Uncertainty Improves Robustness: Moving from deterministic to probabilistic cross-modal fusion acts as a powerful regularizer. By acknowledging what it doesn't know, the model avoids overfitting to the training domain and handles out-of-distribution data much more gracefully.
  • Bidirectional Fusion is Crucial: Letting the visual features refine the text embeddings (and vice versa) creates a more stable joint space for dense prediction than simply using text as a static query or modifier.
  • Soft Targets Prevent Feature Collapse: In medical imaging datasets, different images often share highly overlapping descriptions (e.g., "a round polyp located in the center"). Standard hard contrastive losses force the model to push these representations apart, which harms semantic learning. Soft contrastive targets preserve these nuanced similarities.

Key Results

  • Superior Data Efficiency: When trained on only 10% of the data, MedCLIPSeg achieved an average Dice Similarity Coefficient (DSC) of 81.10%, outperforming the state-of-the-art text-driven model CAT-Seg (78.76%) and vastly outperforming standard UNet (60.95%).
  • Robust Domain Generalization: Under severe domain shifts, MedCLIPSeg exhibited the smallest performance drop (10.0 points), compared to CAT-Seg (11.5), CLIPSeg (15.7), and LViT (18.3).
  • Highly Reliable Calibration: The probabilistic formulation cut the model's Brier Score (a metric for prediction calibration) roughly in half—dropping from ~24% in the deterministic baseline to ~11% across both ID and OOD data.
Visualizations showing that predicted uncertainty peaks precisely along ambiguous lesion boundaries.

Qualitatively, the generated uncertainty maps highly correlate with actual pixel-level segmentation errors (Pearson correlation > 0.92), providing an intuitive visual guide for clinicians to identify where the model might be failing.

Limitations and Future Work

  • Computational Overhead: Generating stable uncertainty maps via Monte Carlo sampling typically requires 30 forward passes at test time, which increases inference latency. However, the authors note that reducing this to 5–10 samples only marginally degrades performance and is sufficient for real-time applications (~25 FPS).
  • 3D Segmentation: While the paper demonstrates a successful proof-of-concept extension to 3D segmentation on the CHAOS CT Liver dataset (achieving a DSC of 88.72% using a 3D VLM backbone), a more comprehensive evaluation and optimization for volumetric data is left for future work.

Summary updated

Abstract

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
BibTeX
@misc{koleilat2026medclipseg,
  author = {Taha Koleilat and Hojat Asgariandehkordi and Omid Nejati Manzari and Berardino Barile and Yiming Xiao and Hassan Rivaz},
  title = {{MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation}},
  year = {2026},
  eprint = {2602.20423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20423},
}

Loading related content...

Figure