Abstract
Medical image registration is a fundamental task in clinical AI, traditionally driven by voxel-intensity metrics. However, these metrics often lack semantic awareness, leading to suboptimal alignments in complex anatomical regions. This paper explores whether "off-the-shelf" features from large-scale vision foundation models—such as DINOv2 and the Segment Anything Model (SAM)—can enhance medical registration. By utilizing these features as auxiliary guidance within a B-spline Free-Form Deformation (FFD) framework, the authors demonstrate improved registration accuracy in cardiac MRI without the need for task-specific retraining. The results suggest that even models trained on natural images possess transferable semantic knowledge beneficial for medical imaging tasks.
Introduction
Image registration—the process of aligning multiple images into a single coordinate system—is essential for longitudinal studies, atlas construction, and multi-modal fusion. Traditional iterative registration typically relies on intensity-based similarity measures like Mean Squared Error (MSE) or Normalized Cross-Correlation (NCC). While effective for local alignment, these methods are prone to "local minima" because they do not understand the underlying anatomy; they only see pixel values.
Deep learning has offered solutions through learned metrics, but these often require extensive ground-truth data or expensive retraining for every new organ or modality. This research investigates a middle ground: leveraging the rich, high-dimensional representations of frozen, pre-trained vision foundation models to guide the registration process.
Methodology
Vision Encoders
The researchers integrated three distinct Vision Transformer (ViT) encoders into a standard iterative registration framework:
- DINOv2: A self-supervised model trained on 142 million natural images.
- SAM (Segment Anything Model): A supervised model trained on 11 million natural images for promptable segmentation.
- MedSAM: A specialized version of SAM fine-tuned on 1.6 million medical images.
Registration Workflow
The registration process follows an iterative optimization loop using a B-spline Free-Form Deformation (FFD) model, which allows for non-rigid, flexible warping. The workflow consists of:
- Feature Extraction: Both the "fixed" and "moving" images are passed through the frozen encoders.
- Dissimilarity Metrics: The difference between feature maps is calculated using either L1 distance or Cosine Similarity.
- Optimization Variants: The study tested two primary configurations. Variant 1 used only the encoder features to drive the registration. Variant 2 used encoder features as an auxiliary "guidance" term alongside a traditional NCC intensity metric.
Experimental Setup
The framework was evaluated using the ACDC (Automated Cardiac Diagnosis Challenge) dataset, comprising cardiac cine MRI scans from 150 patients. The specific task involved registering the heart at its most expanded state (end-diastolic) to its most contracted state (end-systolic). Performance was measured by the overlap (Dice Score) and boundary distance (Hausdorff Distance) of anatomical segmentations, including the Left Ventricle, Right Ventricle, and Myocardium.
Key Insights and Results
The Superiority of Cosine Similarity
A critical technical finding was that Cosine Similarity outperformed L1 distance for comparing feature maps. It provided a smoother optimization landscape and a wider "capture range," allowing the algorithm to successfully align images even when the initial displacement was significant.
Guidance vs. Replacement
The experiments revealed that foundation model features are most effective when used as guidance rather than a total replacement for intensity metrics. While Variant 1 (features only) struggled to capture fine-grained pixel details, Variant 2 (NCC + Feature Guidance) consistently outperformed the baseline. This indicates that foundation models provide the necessary semantic context (the "what" and "where" of organs), while traditional metrics provide the "pixel-perfect" precision required for medical accuracy.
Natural vs. Medical Domain Transfer
While MedSAM (the medical-specific model) achieved the highest accuracy, DINOv2—despite being trained exclusively on natural images—was remarkably competitive. This suggests that the fundamental visual primitives learned from natural images (edges, textures, and geometric shapes) are highly transferable to the medical domain.
Quantitative Performance
- Accuracy: The baseline NCC-only approach achieved a mean Dice Score (DSC) of 0.836. Adding MedSAM guidance increased this to 0.845.
- Boundary Precision: The 95th percentile Hausdorff Distance (HD95) improved significantly, dropping from 1.916 in the baseline to 1.711 with DINOv2 guidance.
- Plausibility: Feature-guided registration resulted in more physically plausible deformation fields with fewer instances of "folding" or unrealistic warping.
Technical Considerations: Non-Commutativity
The authors highlighted a subtle but vital implementation detail: the non-commutativity of warping and encoding. Specifically, extracting features from a warped image yields different results than warping the features extracted from the original image. For maximum accuracy, the image should be warped in the image space before being passed through the encoder in each iteration.
Limitations and Future Directions
Despite the promising results, several challenges remain:
- Dimensionality: Most foundation models are 2D-native, whereas medical data is 3D. Processing 3D volumes as 2D slices is computationally heavy and ignores cross-slice spatial context.
- Computational Cost: Running large ViT encoders within an iterative loop increases the time required for registration compared to simple intensity metrics.
- Spatial Resolution: ViT encoders typically process images in "patches," resulting in lower-resolution feature maps that may discard fine anatomical details.
Future work will likely focus on the integration of 3D-native medical foundation models and the exploration of intermediate encoder layers to capture multi-scale anatomical information.
Summary updated