CAST: Calibrating Correspondences for Training-Free Biomedical In-Context Segmentation

Mingze Jiang, Jiashu Han, Haonan Chen, Zhiyi Li, Yilun Du, Kunzan LiuarXiv 2026 · arXiv

TL;DR

CAST (Calibrating Correspondences) is a training-free framework that enables high-performance biomedical image segmentation using only a few annotated examples at inference time. By leveraging a frozen DINOv3 backbone and introducing novel "stability" and "coherence" tests to calibrate feature matching, it outperforms the strongest baselines by over 12 Dice points across diverse 2D and 3D benchmarks.

What Problem Does This Paper Address?

Biomedical image segmentation is notoriously "brittle." A model trained to segment livers on one specific CT scanner often fails when faced with a different scanner, a new imaging protocol, or a different patient population. Traditionally, fixing this requires gathering dense expert annotations for the new domain and retraining the model—a process that is both slow and expensive.

In-context segmentation (ICS) offers a more flexible alternative: instead of retraining, you provide the model with a few "exemplars" (images + masks) at test time to define the task. However, existing ICS methods often struggle with medical images. Unlike natural images, where a bird stands out clearly against a sky, biomedical structures often share nearly identical textures and intensities with their neighbors. As shown in Figure 2, standard vision foundation models like DINOv3 produce "blurry" similarity maps in medical contexts, making it difficult to distinguish between visually similar but anatomically distinct regions.

Figure 2. Comparison of DINOv3 feature similarity between natu- ral and biomedical images. Similarity maps are obtained by com- puting cosine similarities between query patch features and an- notated target features from the source image. Natural images ex- hibit clear similarity contrast between foreground and background, whereas biomedical images show similar responses across neigh- boring anatomical structures. Green contours indicate source an- notations.

Methods

The authors introduce CAST, a training-free framework built on top of a frozen DINOv3 backbone. Because they don't update the model weights, the "intelligence" lies in how they calibrate the correspondence between the source exemplars and the target query image.

The core of CAST consists of two complementary tests:

  1. The Stability Test: The authors apply controlled geometric transformations (rotations, scaling, shifts) to the source images to observe how the underlying features change. This allows them to perform Global Debiasing (suppressing feature dimensions that are too sensitive to noise/geometry) and Local Calibration (weighting features based on how reliably they represent specific anatomical patches).
  2. The Coherence Test: This test moves beyond individual pixels to look at the "neighborhood." It evaluates candidate matches based on Feature Contrast (do the surrounding textures match?) and Layout Consistency (is the spatial arrangement of "target" vs. "non-target" regions consistent with the source mask?).

These tests are integrated into a refinement framework where initial candidates are retrieved via simple similarity and then reranked using the stability and coherence metrics before a final voting stage produces the segmentation mask.

Figure 3. Overview of CAST. Given an annotated source image, its mask, and an unlabeled query image, CAST calibrates and refines source–query patch correspondences through two complementary tests. The stability test characterizes feature variations induced by geometric transformations, providing global debiasing before candidate retrieval and patch-specific metrics for subsequent reranking. The coherence test evaluates retrieved candidates through neighborhood feature contrast and target–non-target layout consistency. Within the correspondence refinement framework, globally debiased features guide class-conditional candidate retrieval, after which source-specific metrics and neighborhood evidence jointly rerank the candidates. The top-Ntop candidates from each class are then aggregated through voting to produce the final segmentation.

Datasets

The researchers evaluated CAST on eight diverse biomedical benchmarks:

  • 2D Datasets: Chest X-ray (lungs), REFUGE (optic disc/cup), GlaS (glands), and DMBM (fluorescence microscopy of brain regions).
  • 3D Datasets: CTPelvic1K (pelvic structures), LPBA40 (brain MRI), CHAOS (liver), and AMOS22 (abdominal organs).

The evaluation covered challenging scenarios including cross-subject transfer, cross-marker transfer in microscopy, and cross-modality transfer (e.g., using an MRI exemplar to segment a CT scan).

Key Insights and Ideas

  • Visual Similarity is a Trap: In medical imaging, looking similar isn't enough. Because adjacent tissues often have identical appearances, a model must rely on structural context and feature reliability to make the right call.
  • Geometry as a Calibration Tool: By "jittering" the source image with synthetic transforms, CAST learns which parts of the feature vector are robust and which are just noise. This effectively "tunes" a general-purpose model for medical data without a single gradient update.
  • Context is King: The "Layout Consistency" check is particularly clever—it uses the provided source mask not just to identify the target, but as a spatial prior to verify that the surroundings of a candidate match make sense.

Key Results

CAST demonstrates a significant leap in performance for training-free methods:

  • Quantitative Dominance: It improved the overall average Dice score by 12.34 percentage points over the strongest baseline (SegGPT) and ranked first across all six volumetric (3D) settings.
  • Annotation Efficiency: On the CHAOS dataset, CAST achieved higher accuracy with just 3 annotated slices than competing methods did with 10 (see Figure 5).
  • Speed: Despite the extra calibration steps, CAST is computationally efficient, with an average inference time of 0.184s per slice, making it faster than many other training-free alternatives.
  • Atlas Label Transfer: The authors showed that CAST can successfully transfer labels from a standardized brain atlas to specific experimental specimens, a task that usually requires complex, slow image registration.
Figure 5. Effect of annotation budget on performance. CAST reaches high accuracy with very few slices.

Limitations and Future Work

While CAST is highly effective, it still relies on the quality of the underlying frozen backbone (DINOv3). In extremely difficult cross-modality cases (like CT to MRI on the AMOS22 dataset), the performance, while better than baselines, still leaves room for improvement.

The authors suggest that future work could explore more complex 2D-to-3D segmentation (using 2D slices to segment a full 3D volume) and expanding the framework's use in large-scale atlas-based applications where manual delineation is currently a major bottleneck.

Summary updated

Abstract

Biomedical image segmentation must keep up with new anatomical structures, subjects, imaging modalities, and acquisition protocols, yet dense expert annotations are scarce and models trained on one cohort often struggle to transfer. In-context segmentation (ICS) offers a way out: a new structure is specified by a few annotated exemplars at inference time. However, reliable correspondence remains challenging in biomedical images, where adjacent structures often share similar appearances, and variations across subjects and imaging conditions further obscure their distinctions. We introduce CAST, a training-free framework built on a frozen DINOv3 backbone to calibrate fine-grained correspondences. Its stability test improves feature reliability at both global and patch-specific levels, while its coherence test assesses whether candidate correspondences are supported by consistent local structural context. Together, these tests enable more reliable correspondence across diverse anatomical structures and imaging conditions. Across eight biomedical benchmarks spanning 2D and 3D segmentation, cross-subject, and cross-modality transfer, CAST outperforms the strongest overall baseline by 12.34 percentage points in Dice and ranks first across all six volumetric settings. The versatility of CAST further opens up opportunities for challenging biomedical applications, including 2D-to-3D segmentation and atlas label transfer.
BibTeX
@misc{jiang2026cast,
  author = {Mingze Jiang and Jiashu Han and Haonan Chen and Zhiyi Li and Yilun Du and Kunzan Liu},
  title = {{CAST: Calibrating Correspondences for Training-Free Biomedical In-Context Segmentation}},
  year = {2026},
  eprint = {2611.22995},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22995},
}
Figure