Abstract
Deformable registration is a fundamental task in medical image processing, aiming to achieve precise alignment by establishing nonlinear correspondences between images. Traditional methods offer good adaptability and interpretability but are limited by computational efficiency. Although deep learning approaches have significantly improved registration speed and accuracy, they often lack flexibility and generalizability across different datasets and tasks. In recent years, foundation models have emerged as a promising direction, leveraging large and diverse datasets to learn universal features and transformation patterns for image registration, thus demonstrating strong cross-task transferability. However, these models still face challenges in generalization and robustness when encountering novel anatomical structures, varying imaging conditions, or unseen modalities. To address these limitations, this paper incorporates Sharpness-Aware Minimization (SAM) into foundation models to enhance their generalization and robustness in medical image registration. By optimizing the flatness of the loss landscape, SAM improves model stability across diverse data distributions and strengthens its ability to handle complex clinical scenarios. Experimental results show that foundation models integrated with SAM achieve significant improvements in cross-dataset registration performance, offering new insights for the advancement of medical image registration technology.
Introduction
Deformable registration—the process of finding a non-linear mapping to align two medical images—is a cornerstone of medical AI. While traditional methods are highly interpretable and adaptable, they are computationally slow, often taking minutes or hours per pair. Deep learning (DL) methods have revolutionized the field by providing near-instantaneous registration. However, these models are notoriously "brittle": a model trained on brain MRIs from one hospital often fails when presented with lung CTs from another or even images from the same organ with different acquisition parameters.
Recently, "foundation models" have emerged to create universal registration tools trained on massive, diverse datasets. However, even these models struggle with robustness when encountering novel anatomical structures or varying imaging conditions. The authors identify that standard training often lands in "sharp" minima—regions of the loss function where a tiny change in data distribution leads to a massive drop in performance. This paper introduces Sharpness-Aware Minimization (SAM) into medical image registration foundation models to solve this "generalization gap."
Methodology
The researchers build their approach on GradICON, a state-of-the-art foundation model architecture that uses a multi-step, multi-scale approach utilizing UNets to predict displacement fields. The core innovation is the integration of SAM into the training loop to optimize for a flatter loss landscape.
Sharpness-Aware Minimization (SAM)
Instead of just looking for the lowest loss point, SAM searches for parameters where the loss is low throughout an entire neighborhood. This effectively seeks out "flat" valleys in the loss landscape rather than narrow, sharp pits. The model minimizes a minimax objective: it tries to find weights that perform well even when subjected to the "worst-case" small perturbation. Mathematically, this encourages the model to find solutions that are less sensitive to the specific noise or distribution shifts of the training data.
Loss Functions and Regularization
To ensure the registration is both accurate and physically plausible, the authors utilize a combination of:
- Localized Normalized Cross-Correlation (LNCC): To measure image similarity.
- Gradient Inverse Consistency Regularizer: To ensure the transformation is mathematically sound, reversible, and prevents the "folding" of voxels (where the transformation overlaps itself).
Experimental Setup
To test true generalization, the model was trained and evaluated on a highly diverse "composite" dataset:
- Training Data: A mix of intra-patient and inter-patient data including COPDGene (Lung CT), OAI (Knee MRI), HCP (Brain MRI), and L2R-Abdomen (Abdomen CT).
- Validation/Testing Data: To prove cross-dataset generalization, the model was tested on unseen benchmarks including ACDC (Cardiac MRI), SLIVER (Liver MRI), and several Learn2Reg challenge datasets (Hippocampus MRI, Abdomen MRCT, and Lung CT).
Key Insights
Flatness Equals Generalization
Visualization of the loss landscape reveals that models trained with SAM have a visibly smoother, flatter loss surface compared to the baseline. This flatness is the mathematical reason the model remains stable when moving between disparate anatomical regions, such as from a brain scan to a cardiac scan.
Diffeomorphic Integrity
In medical registration, "folding" is a physical impossibility. The study found that optimizing for sharpness-awareness naturally led to fewer "folded voxels" (Jacobian determinant $\leq 0$). This results in more biologically plausible deformations without sacrificing alignment accuracy.
Foundation Model Synergy
SAM appears particularly effective when applied to foundation models. Because these models already see diverse data, SAM ensures the model does not "over-specialize" on any single feature of that diverse data, preserving its universal applicability.
Results
The proposed method was compared against traditional methods (Elastix), CNN-based models (VoxelMorph), and Transformer-based models (TransMorph).
- Superior Accuracy: On the ACDC (Cardiac) dataset, the model achieved a Dice score of 0.7360, outperforming all baselines. On the SLIVER (Liver) dataset, it reached 0.8722.
- Robustness to Folding: In the Liver MRI task, the SAM-integrated model was the only one to achieve zero non-positive Jacobian determinants, indicating perfectly smooth, fold-free registrations.
- Cross-Modality Success: The model maintained high performance across five diverse datasets (Cardiac, Hippocampus, Abdomen, Lung, and Liver) without needing task-specific fine-tuning, outperforming the original uniGradICON foundation model.
Conclusion and Future Work
Integrating SAM into medical image registration foundation models significantly improves their ability to generalize to unseen clinical data while maintaining physical plausibility. While SAM requires two forward/backward passes per iteration—increasing training time—inference speed remains unchanged, making it a viable strategy for high-performance clinical tools.
Future work will explore combining SAM with hybrid Transformer-CNN architectures and validating the model's performance in specific downstream clinical tasks, such as tumor tracking and surgical navigation. The code is available at https://github.com/Promise13/fm_sam.
Summary updated