Keeping Capture Range When Registration Networks Learn New Modalities

Learned image registration networks align two scans, such as an MRI and a CT of the same abdomen, in a single forward pass. Hospitals do not hand these networks one fixed dataset. They add scanners and modality pairs over time, for example MR-to-CT first, then CT-to-cone-beam-CT, then MR-to-ultrasound. When the network is fine-tuned on each new pair, it forgets the old ones. We think the first thing it forgets is not fine alignment of scans that already start close together. It is capture range: the ability to recover when the two scans start far apart, rotated or shifted by a large amount, where a local optimizer falls into a wrong alignment. Standard evaluations hide this, because they score registration only from roughly pre-aligned starting points.

We propose to measure capture-range forgetting directly and to counter it with a replay scheme that costs almost nothing. The scheme works as follows. The network reads modality-invariant structural maps, meaning edge and gradient-orientation images that look the same whether they came from CT or MRI. Old tasks are rehearsed by taking synthetic structural maps, warping them with large known transformations, and training the network to undo those warps. The ground-truth answer is exact, no second modality has to be generated, and no patient scan has to be stored. As a secondary contribution, we account for how many similarity evaluations a classical global search needs to restore the same capture range. That count is precisely the quantity that quantum minimum-finding would speed up, so it gives an honest, falsifiable estimate of whether quantum dimensional speedups touch the real bottleneck in registration.

Core Claim

Under sequential modality shifts, a learned multimodal registration network loses robustness to large initial misalignment faster than it loses near-aligned accuracy. Rehearsing large-transform pairs built from cheap synthetic structural maps should recover most of that lost robustness. It should do so at a small fraction of the cost of diffusion-based generative replay, and at lower per-scan runtime than putting a classical global search in front of the network.

The idea is new in three respects relative to the file. First, continual learning has not been studied for multimodal registration. LADDA (LADDA) uses a frozen, pre-trained latent diffusion model as an anatomy prior and is trained offline. MMRegNet (Unsupervised Multi-modality Registration Network Based on Spatially Encoded Gradient Information) and GPO (Gaussian Primitive Optimized Deformable Retinal Image Registration) are likewise trained once on a static dataset. Second, the generative-replay work in the file replays task inputs themselves, which is where its cost comes from. Lifelong Domain Adaptive 3D Human Pose Estimation replays 2D poses with a diffusion sampler. Prioritized Generative Replay replays RL transitions with a conditional diffusion model. We instead replay the transformation-recovery skill, using inputs that need no faithful anatomy. Third, the evaluation axis is capture range under forgetting, not accuracy alone.

Research Program

The network has two parts. A per-modality encoder maps each scan to a structural map. We build this on multi-scale gradient-orientation consistency, following the gradient encoding in MMRegNet, because tissue boundaries stay in the same place across modalities even when intensities invert. A shared displacement head predicts a rigid-plus-affine initialization followed by a deformable field.

When a new modality pair arrives, we train the new encoder and fine-tune the shared head on real pairs. Interleaved with these real pairs are replayed pairs. Each replayed pair is a structural map $S$ together with its warped copy $S \circ T$, where $T$ is a transformation sampled with deliberately large rotations (up to 90°) and translations. The head is supervised to recover $T$ exactly.

Replayed structural maps come from one of three sources, and comparing them is the core ablation. The first source is a small single-pass generator, a variational autoencoder trained on structural maps of past tasks. It is fine-tuned at task boundaries, but it operates at low resolution, so it sidesteps the iterative-denoising cost that the replay literature identifies as the bottleneck (Wu et al., Gao et al.). The second source is a procedural generator of random tubular and blob structures that contains no anatomical content at all. The third, used only as a reference, is a diffusion generator in the style of the pose-estimation work.

To measure capture range, we register each test pair from initial misalignments of increasing size. Let $s(\theta)$ be the fraction of test pairs whose post-registration Dice exceeds a fixed threshold when the starting rotation is $\theta$ (with proportional translation). We summarize capture range as

$$\mathrm{CR} = \frac{1}{\theta_{\max}} \int_0^{\theta_{\max}} s(\theta)\, d\theta,$$

with $\theta_{\max} = 90^\circ$. Forgetting is the drop in $\mathrm{CR}$ on task $k$ after training on later tasks, reported separately from the drop in Dice at $\theta = 0$.

The quantum accounting is an analysis, not an implementation. A classical global initializer evaluates a similarity score over $N$ discretized pose hypotheses and keeps the best. Quantum minimum-finding, the subroutine Rebentrost et al. use to compress Hedge's dependence on the number of strategies from $O(N)$ to $O(\sqrt{N})$ (Quantum algorithms for hedging and the learning of Ising models), would need on the order of $\sqrt{N}$ oracle calls for the same search. We will report the $N$ needed to match the capture range our replayed network achieves, and the share of total registration compute this search represents. We will also state the limits of the other quantum results in the file.

Two quantum results do not cover this setting. The polylog-dimension zeroth-order rates in Fast Zeroth-Order Convex Optimization with Quantum Gradient Methods assume convexity, which registration violates. The Schrödinger wave-packet method in Quantum algorithms for escaping from saddle points certifies only second-order stationary points, not escape from false minima. Both require fault-tolerant hardware. The output is therefore an idealized upper bound on quantum benefit, which tells the field whether the bottleneck it targets is the one registration actually has.

Baselines

The naive reference is sequential fine-tuning with no protection against forgetting. Its counterpart is a joint-training upper bound that sees all tasks at once. Both use the same architecture, data, and number of gradient steps.

The cheapest serious competitor is parameter regularization, such as elastic weight consolidation on the shared head. The file reports that generative replay is often far less efficient than regularization (Gao et al.). We tune its strength with the same validation budget as our replay ratio.

The strongest replay competitor is diffusion-based generative replay of past images, following the domain-aware diffusion memory of Lifelong Domain Adaptive 3D Human Pose Estimation. That paper reports that removing the diffusion memory costs roughly 8 mm MPJPE, which is the strongest file evidence that input replay works. We run it under an equal wall-clock budget as well as at its native cost.

The strongest non-learning competitor is a classical GPU-accelerated optimizer with a coarse global stage. ConvexAdam achieved the best overall rank in Learn2Reg; no reported numeric result was found on Lacuna for ConvexAdam on specific tracks. We run it from the same perturbed starts.

For static, per-task accuracy references we use the following reported numbers. MMRegNet reports Dice of 79.00±8.06% for T1→CT liver registration on CHAOS and 80.28±7.22% for the left-ventricle cavity on MM-WHS (MMRegNet). LADDA reports DSC 92.5±1.56 on an in-house MR-to-CT abdominal set (LADDA). That set is not public, so we treat the number as context, not a target. No reported result was found on Lacuna for continual multimodal registration.

Beating the baselines means two things. Our method must retain at least as much $\mathrm{CR}$ as diffusion replay at under a tenth of its update cost. It must also exceed regularization in $\mathrm{CR}$ while matching it at $\theta = 0$.

The key ablation swaps the anatomical generator for the procedural one. A second ablation keeps the replay inputs but samples only small transforms, which isolates whether large-transform rehearsal specifically drives capture-range retention.

Why not just…?

Why not just use elastic weight consolidation instead of any replay? It is cheap, and the file supports skepticism of replay cost. However, regularization protects weights in proportion to their importance on the old data seen during training. If old training data rarely contained large misalignments, the weights that matter for capture range are never marked as important. Our metric will show whether regularization preserves $\theta = 0$ Dice while letting $\mathrm{CR}$ collapse. If it preserves both, the replay contribution fails and we will say so.

Why not just put a classical global search before the network, so capture range never matters? This is the real threat. The MMD-based registration of Crane et al. shows that smoothed objectives widen the basin but still behave as local optimizers. GoTS (Liu et al.) cuts the rigid search to three dimensions, yet still needed 25.9 seconds on a 300k-point scene, and it applies only to rigid point clouds. Exhaustive pose search on 3D volumes multiplies the per-scan cost by $N$. Our experiments report accuracy and runtime for search-plus-network against replayed-network-alone. If the search wins on both, the practical recommendation becomes "search first", and the quantum accounting becomes the main result, because that search is the step quantum minimum-finding addresses.

Why not just store a small buffer of real past scans? Where privacy rules allow it, a buffer is the simplest option, and we include it as an oracle-like reference. Gap 3 of the file notes that multi-center storage of patient scans is often not permitted, which is the setting we target. The procedural-generator ablation tests whether even synthetic anatomy is needed.

Proposed Experiments

The task sequence uses public data: Learn2Reg Abdomen MR-CT, then CHAOS T1/T2→CT, then MM-WHS cardiac CT-MR, then Learn2Reg CURIOUS MR-ultrasound. This gives a realistic accumulation of modality pairs and anatomies. We will also test two other orders to check order sensitivity.

For every method and order, we report the following on every past task after each new one: Dice and average symmetric surface distance at $\theta = 0$, $\mathrm{CR}$ over 0–90°, folding measured as the standard deviation of the log Jacobian determinant, update wall-clock per task boundary, and registration wall-clock per scan.

The first decisive check is whether capture-range forgetting exceeds near-aligned forgetting under naive fine-tuning. If it does not, the premise is wrong.

The second check is whether procedural replay matches anatomical replay. If it does, the result is a stronger and cheaper finding: the skill of recovering transformations transfers without anatomy.

The third check is the equal-runtime comparison against ConvexAdam with a global stage, together with the measured $N$ and its share of compute. Taken together, these determine whether the remaining bottleneck is learned forgetting or global pose search. If it is pose search, the analysis bounds what an idealized $\sqrt{N}$ quantum search could recover.