Warm-Starting Distributed Registration with Learned Deformations Without Losing What the Network Knew

Image registration means finding the smooth warp that aligns one 3D scan onto another. Two families of methods compete, and each fails differently. Neural networks predict the warp in one forward pass. When trained with anatomical labels, they align brain structures better than classical optimizers on data like their training set, but they break down on scans from other sites. Classical iterative optimizers start each scan pair from scratch, are slower, and are capped by what image intensities alone can reveal, yet they hold up on unfamiliar data. At gigavoxel resolution, a recent distributed system splits each volume across GPUs and makes iterative registration feasible in about a minute. That system still starts every pair from an identity or affine transform. No network can run on such volumes at full resolution anyway.

We propose to run a label-trained network at coarse resolution on one GPU, upsample its predicted warp, and use it as the starting point for the distributed fine-resolution optimizer. During refinement, a confidence-weighted penalty keeps the solution close to the network's prediction where the network is reliable, and lets the intensity objective take over where it is not. The goal is to keep the network's in-distribution accuracy, the optimizer's out-of-distribution robustness, and fewer of the expensive multi-GPU iterations, all at once.

Core Claim

The obvious combination is to initialize an iterative optimizer with a network's prediction and let it run. We expect that combination to fail in an informative way.

Deep Learning in Medical Image Registration: Magic or Mirage? benchmarks classical optimizers (ANTs, Greedy, NiftyReg, FireANTs) against learned models. It reports supervised networks at Dice near 0.90 versus about 0.80 for classical tools on OASIS. The authors attribute the advantage to label priors that lie beyond what intensity similarity can reward, and call this the "mutual-information ceiling." If so, an optimizer that minimizes only an intensity loss has every incentive to undo exactly that advantage: a warm start followed by unconstrained refinement should drift back toward the classical solution. The same paper finds the order reverses on external datasets (LPBA40, MGH10, IBSR18, CUMC12), where classical methods beat the transferred networks.

Our claim is that a single refinement objective resolves this tension. The objective treats the network's warp as a prior whose strength varies spatially, set by an estimate of the network's local reliability. The resulting method should match the supervised network in distribution, match or beat classical optimization out of distribution, and reach the classical method's final loss in a fraction of the distributed iterations. The paper "Magic or Mirage?" motivates this work by documenting the gap. It explicitly leaves the hybrid untested and works only at clinical resolution.

Research Program

The substrate is FFDP from A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration. FFDP splits the image volumes and the warp field across a grid of GPUs. It keeps each GPU's boundary region (the "halo") synchronized so that local similarity terms such as local normalized cross-correlation stay correct across seams. It also uses a ring scheme to fetch intensities that live on other GPUs. FFDP registered an 11.8-billion-parameter ex-vivo brain problem in about 58 seconds on 8 A6000 GPUs, starting from identity or affine. We reuse its kernels unchanged; what we add is a new starting point and a new term in its objective.

The refinement objective over the warp $\varphi$ is

$$E(\varphi) = D(I_f, I_m \circ \varphi) + \lambda R(\varphi) + \mu \sum_x c(x)\, \lVert \varphi(x) - \varphi_0(x) \rVert^2 .$$

Here $I_f$ and $I_m$ are the fixed and moving images, and $D$ is the intensity dissimilarity FFDP already computes (local cross-correlation or mutual information). $R$ is its smoothness regularizer, with weight $\lambda$. $\varphi_0$ is the network's coarse warp, upsampled to full resolution. $c(x) \in [0,1]$ is a per-voxel confidence, and $\mu$ is the global strength of the prior. Every term is local, so the new term splits across GPUs with no added communication beyond the existing halos.

We estimate confidence $c$ from disagreement across a small ensemble of independently trained models, or from test-time flips of the input. The idea is that disagreement should rise on unfamiliar anatomy or contrast. Calibrating $c$ is the main scientific risk. We therefore also test a cheaper estimate: the local similarity gain of $\varphi_0$ over the affine transform.

Modular design appears here as a narrow interface, not as a goal in itself. The amortized predictor emits a coarse field together with a confidence map. A resharding module scatters both into FFDP's grid layout. The solver consumes them through the added penalty term. Defining this boundary cleanly lets us swap predictors (TransMorph, LapIRN, SynthMorph, as benchmarked in "Magic or Mirage?") without touching the fused kernels. That addresses the concern in Section 1.2 of the research file that abstraction layers block kernel fusion.

The meta-learning connection is direct. Initializing Bayesian Hyperparameter Optimization via Meta-Learning warm-starts a search by retrieving the configurations that worked best on similar past datasets, and it reports reaching in a few steps what cold search misses after 50 iterations. We apply the same logic to the optimizer's state rather than its hyperparameters, and we borrow that line of work's central worry, harm from transferring a bad prior. The confidence term is our guard against that harm. Those papers and their successors, Efficient and Robust Automated Machine Learning and Auto-Pytorch, supply the framing of retrieval and portfolio warm starts. They do not supply a mechanism that works on dense spatial fields.

Baselines

The do-nothing reference is FFDP, or FireANTs where volumes fit on one GPU, run cold. It uses its default affine initialization and its full coarse-to-fine resolution pyramid, with the same intensity loss, regularizer, and iteration schedule we use. This baseline is important because the pyramid's coarse levels are already a warm start, built from the optimizer's own low-resolution solution. On OASIS, the reported figure for classical tools is Dice about 0.80 ("Magic or Mirage?"). On the external datasets, classical methods are reported to beat OASIS-trained networks, but no reported per-dataset number was found on Lacuna, so we will rerun these values ourselves.

The strongest existing competitor in distribution is the best supervised network from "Magic or Mirage?" used alone, reported at Dice near 0.90 on OASIS. We train it on the same OASIS split with the same labels and evaluate it at the same coarse resolution we use for warm-starting. We also evaluate it at native resolution wherever memory allows, so that coarse prediction does not unfairly handicap it.

The strongest naive hybrid is the same network's warp used as an initialization with $\mu = 0$, meaning unconstrained refinement. This is the ablation that isolates our key ingredient. A second ablation fixes $c(x)$ at a constant, which tests whether spatial confidence matters or a global trust weight suffices. A third baseline borrows from the meta-learning tradition: initialize from the composed warp of the most similar training subject, retrieved by image descriptors in the spirit of Initializing Bayesian Hyperparameter Optimization via Meta-Learning. This checks whether a learned predictor is needed at all, or whether retrieval is enough.

All methods share the data splits, the regularizer, the GPU count, and a hyperparameter budget. The budget is a fixed number of trials on a held-out OASIS validation set, applied to $\lambda$ for every method and additionally to $\mu$ for ours. External datasets are never used for tuning.

Beating the baselines means three things together. In distribution, the method must come within noise of the supervised network while the $\mu = 0$ hybrid falls measurably toward the classical level. Out of distribution, it must be no worse than cold FFDP or FireANTs on all four external datasets and better than the network alone. In speed, it must reach cold FFDP's final dissimilarity value in substantially fewer fine-level iterations.

Why not just…?

Why not just warm-start with the network and refine without any prior term? This is the default, and it is the real threat. If the label-prior gains survive unconstrained refinement, our added term is unnecessary. The mutual-information-ceiling argument in "Magic or Mirage?" predicts they will not survive, because the intensity loss does not reward label-consistent deformations that intensity cannot see. That is a prediction, not a measurement. The $\mu = 0$ ablation settles it directly. If it matches our method, we will report the simpler result: warm starts preserve the gains and only reduce cost.

Why not just rely on the optimizer's own coarse-to-fine pyramid, which is already a warm start? For speed, this may be nearly as good. The coarse classical solution is cheap to compute and already removes most of the large-scale misalignment. Its starting point, though, is bounded by intensity information, so it cannot carry label priors into the fine levels. We therefore expect the pyramid to match us on time-to-loss more closely than on Dice. We report both measures separately, so a reader can see whether the amortized initialization buys accuracy, speed, or neither.

Why not just use the network alone, or a foundation model, and skip iteration? The network-alone baseline measures this directly. The file's evidence argues against it. Learned registration degrades on external brain datasets in "Magic or Mirage?". In segmentation, Auto-nnU-Net reports task-specific search beating MedSAM2 on 9 of 10 Medical Segmentation Decathlon tasks. At gigavoxel scale the network cannot run at full resolution in any case, so some fine-scale optimization is unavoidable.

Proposed Experiments

At clinical resolution, we train on OASIS T1 scans and test in distribution on held-out OASIS pairs. We test out of distribution on LPBA40, MGH10, IBSR18, and CUMC12, scoring Dice on labeled structures and the fraction of voxels where the warp folds over itself. For every method we also plot dissimilarity and Dice against iteration count. Here everything fits in memory, so this experiment cleanly tests the accuracy claim.

At scale, we use the 100 µm to 250 µm ex-vivo brain pair from FFDP on 8 GPUs and report wall-clock time to reach the cold-start final dissimilarity. To obtain labels at scale, we also upsample labeled OASIS pairs to gigavoxel resolution, adding synthetic fine texture. This construction is artificial; it is used only to check that the clinical-scale accuracy ordering survives domain decomposition and coarse-to-fine upsampling of $\varphi_0$.

A calibration study checks whether $c(x)$ actually falls in regions where the network's warp is wrong, measured as local Dice error against labels on the external datasets.

The decisive checks are three. First, the gap between our method and the $\mu = 0$ hybrid in distribution: if there is no gap, the confidence prior is unnecessary. Second, whether our method ever loses to cold classical optimization out of distribution: if it does, the confidence estimate is failing to block harmful transfer. Third, whether the iteration savings at gigavoxel scale exceed those of the optimizer's own pyramid.