Align then Refine: Text-Guided 3D Prostate Lesion Segmentation

Cuiling Sun, Linkai Peng, Adam Murphy, Elif Keles, Hiten D. Patel, Ashley Ross, Frank Miller, Baris Turkbey, Andrea Mia Bejar, Halil Ertugrul Aktas, Gorkem Durak, Ulas BagciarXiv 2026 · arXiv

TL;DR

This paper introduces a text-guided, multi-encoder U-Net architecture that leverages biomedical vision-language models (VLMs) to significantly improve 3D prostate lesion segmentation from biparametric MRI. By combining bottleneck semantic alignment losses with a final-stage, confidence-gated cross-attention refiner and a phase-scheduled training curriculum, the framework achieves new state-of-the-art results on the competitive PI-CAI benchmark.

What Problem Does This Paper Address?

Accurate 3D segmentation of prostate cancer lesions from biparametric MRI (bp-MRI)—which includes T2-weighted (T2W), diffusion-weighted imaging (DWI), and apparent diffusion coefficient (ADC) sequences—is vital for targeted biopsies and treatment planning. However, it remains a notoriously difficult task because lesions are often small, poorly contrasted, and highly heterogeneous.

Existing approaches fall short in distinct ways:

  1. Image-only models (like nnU-Net) rely on fixed, modality-agnostic fusion strategies that struggle to maintain consistency across slices when data is misaligned.
  2. Medical foundation models (like MedSAM2) deliver strong few-shot performance but frequently fail to enforce the volumetric consistency required for fine-grained 3D structures.
  3. Vision-Language Models (VLMs) (like BiomedCLIP) offer helpful semantic priors, but their pretraining focuses on global, image-level text alignment rather than the localized, voxel-wise guidance needed to trace precise lesion boundaries.

This paper bridges the gap by injecting fine-grained text guidance into a 3D multi-encoder architecture without incurring prohibitive computational overhead.

Methods

The authors propose a multi-encoder U-Net that processes each MRI modality through a specialized encoder before fusing them at a shared bottleneck. The core innovation lies in how text features from a frozen BiomedCLIP model (prompted with "prostate lesion") are integrated using an "Align then Refine" paradigm:

  1. Similarity Head & Bottleneck Alignment: At the bottleneck, fused image features are projected into the text embedding space. A temperature-scaled cosine similarity map is computed against the text prompt embedding to create an interpretable lesion-likelihood heatmap.
  2. Dual Bottleneck Losses:
  3. Alignment Loss ($L_{\text{align}}$): A foreground-only objective that maximizes text-image similarity strictly inside the lesion boundaries, preventing background noise from destabilizing early training.
  4. Heatmap Loss ($L_{\text{heat}}$): A binary cross-entropy loss applied over the entire volume to suppress false-positive background activations and calibrate the heatmap.
  5. Final-Stage Cross-Attention Refinement: Instead of applying heavy attention mechanisms throughout the network, a lightweight cross-attention refiner is attached only to the highest-resolution decoder stage. It uses a learnable gate and a confidence threshold ($\tau$) to apply subtle, localized boundary adjustments only where the base network is already highly confident.
  6. Phase-Scheduled Training: To keep optimization stable, training is broken into three distinct phases:
  7. Phase 1: Segmentation-only training to stabilize multimodal fusion.
  8. Phase 2: Semantic transfer via the auxiliary alignment and heatmap losses.
  9. Phase 3: Auxiliaries are turned off, and the gated cross-attention refiner is enabled to polish boundaries.
[T2W, ADC, DWI Encoders] ➔ [Bottleneck + BiomedCLIP Text] ➔ [Decoder] ➔ [Gated Cross-Attention Refiner] ➔ Final Segmentation
                                  ↳ (Dual Alignment Losses)

Datasets

The model was trained and evaluated using the public PI-CAI (Prostate Imaging-Cancer AI) dataset, a large-scale benchmark for prostate cancer detection and segmentation on bp-MRI. The authors utilized patient-level cross-validation, splitting the data into 70% for training, 10% for validation, and 20% for testing.

Key Insights and Ideas

  • Decoupling Semantics from Boundaries: Trying to learn global text-image alignment and fine-grained boundary corrections simultaneously causes optimization conflicts. By forcing semantic alignment at the bottleneck and boundary refinement at the final decoder stage, the network handles both tasks efficiently.
  • The Power of Phase Scheduling: Activating the cross-attention refiner too early degrades performance (dropping the Dice score to 0.7081). Waiting until the feature space stabilizes allows the attention mechanism to act as a precision editor rather than a chaotic disruptor.
  • Foreground-Only Regularization: Because prostate lesions exhibit extreme class imbalance (they are tiny relative to the whole volume), the foreground-only alignment loss provides a stable gradient anchor that prevents the model from simply predicting "background" everywhere.

Key Results

The proposed framework establishes a new state-of-the-art on the PI-CAI dataset, outperforming strong medical imaging baselines across major overlap and surface metrics:

  • Dice Similarity Coefficient: Achieved 0.7326, outperforming the industry-standard nnU-Net (0.7192), MedSAM2 (0.5729), and Swin UNETR (0.5811).
  • Normalized Surface Distance (NSD): Reached 0.7541, demonstrating superior boundary alignment compared to nnU-Net (0.7320).
  • Precision: Improved to 0.7476 (vs. nnU-Net's 0.7339), indicating a substantial reduction in false-positive lesion predictions.

Ablation studies confirmed that combining both the alignment and heatmap losses yielded the best outlier robustness, dropping the 95th-percentile Hausdorff Distance (HD95) significantly before the refinement phase.

Limitations and Future Work

While highly effective, the authors note several areas for improvement:

  • Prompt Sensitivity: The framework relies on a single, fixed text prompt ("prostate lesion"). How the model reacts to alternative phrasing or prompt ensembling remains unexplored.
  • Dataset Diversity: Evaluation was restricted to a patient-level partition of the PI-CAI dataset; broader validation across multiple imaging centers, scanner manufacturers, and clinical protocols is required.
  • Long-Tail Deviations: Although the refiner improves the majority of boundaries, it can occasionally introduce isolated surface errors in exceptionally small lesions, slightly impacting the HD95 metric. Future work will explore boundary-aware loss functions and uncertainty-conditioned gating to mitigate this.

Summary updated

Abstract

Automated 3D segmentation of prostate lesions from biparametric MRI (bp-MRI) is essential for reliable algorithmic analysis, but achieving high precision remains challenging. Volumetric methods must combine multiple modalities while ensuring anatomical consistency, but current models struggle to integrate cross-modal information reliably. While vision-language models (VLMs) are replacing the currently used architectural designs, they still lack the fine-grained, lesion-level semantics required for effective localized guidance. To address these limitations, we propose a new multi-encoder U-Net architecture incorporating three key innovations: (1) an alignment loss that enhances foreground text-image similarity to inject lesion semantics; (2) a heatmap loss that calibrates the similarity map and suppresses spurious background activations; and (3) a final-stage, confidence-gated multi-head cross-attention refiner that performs localized boundary edits in high-confidence regions. A phase-scheduled training regime stabilizes the optimization of these components. Our method consistently outperforms prior approaches, establishing a new state-of-the-art on the PI-CAI dataset through enhanced multi-modal fusion and localized text guidance. Our code is available at https://github.com/NUBagciLab/Prostate-Lesion-Segmentation.
BibTeX
@misc{sun2026align,
  author = {Cuiling Sun and Linkai Peng and Adam Murphy and Elif Keles and Hiten D. Patel and Ashley Ross and Frank Miller and Baris Turkbey and Andrea Mia Bejar and Halil Ertugrul Aktas and Gorkem Durak and Ulas Bagci},
  title = {{Align then Refine: Text-Guided 3D Prostate Lesion Segmentation}},
  year = {2026},
  eprint = {2604.18713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18713},
}

Loading related content...

Figure