Which Parts of a Multimodal Registration Do the Images Actually Determine?

Image registration aligns two pictures of the same thing, such as an MRI and an ultrasound of a brain, or a camera frame and a LiDAR scan of a street. It does this by searching for the spatial transformation that makes one image line up with the other. When the two images come from different sensors, their brightness values cannot be compared directly. The standard fix is mutual information: a measure of how well one image's intensities predict the other's. Registration then searches for the transformation that maximizes it. Mutual information sometimes fails badly. On prostate MRI-to-ultrasound it failed to get below 24 mm median landmark error, while a label-supervised network reached 3.6 mm (Hu et al.). When it fails, the field has no tool for asking why. Was a parameter simply invisible to the images? Did several parameters trade off against each other? Or did the objective have several equally good answers?

This proposal brings in a tool from parameter identifiability, the study of whether a model's parameters can be uniquely recovered from data. The tool is an information-theoretic diagnostic built for structural engineering models (Parameter Identifiability through Information Theory), which we turn into a ground-truth-free identifiability map for multimodal registration. The procedure has three steps. First, apply known random perturbations to the moving image's transformation. Second, re-run the registrator after each perturbation. Third, treat the registrator's output as the "measurement" of the perturbation. For each transformation parameter, the result separates three failure types: blindness, compensation between parameters, and multiple discrete solutions. We then test two things. Does this map predict where real registration errors land? And does acting on it, by freezing blind parameters and tying compensating ones, beat uniform regularization on public multimodal benchmarks?

Core Claim

Practical non-identifiability of registration parameters can be measured at the operating point of any registrator, without labels. The measurement uses mutual information between injected perturbations and recovered estimates. Information measures matter here, rather than local curvature or variance, because the dominant multimodal failures are non-Gaussian. Recovered estimates often split into several modes, as with symmetric anatomy or near-equivalent yaw solutions. Parameters also compensate for each other non-linearly, as with rotation against translation, or adjacent B-spline control points sitting in textureless regions. If the claim is wrong, a plain covariance of the perturbation responses will predict errors equally well, and we treat that outcome as a decisive negative result.

Background and Positioning

The diagnostic comes from Capellari et al.. They place independent priors on physical parameters $\theta$, simulate sensor outputs $Y$, and use kernel density estimates of $I(\theta_i;Y)$ to detect insensitivity. In their 8-story building model, the post-yield stiffness had near-zero information while the structure stayed elastic. They use $I(\theta_i;\theta_j\mid Y)$ to detect compensation. Elastic modulus and moment of inertia showed very high conditional information because only their product enters the stiffness. We reuse this estimator unchanged, but swap the forward model. Their $Y$ is a simulated displacement record. Ours is the output of an image registrator responding to a known perturbation.

The formal identifiability literature constrains the design more than it supplies a method. Dörfler et al. show that generic identifiability in linear causal models is decidable in polynomial space. They also show that deciding identifiability for specific numerical values is $\forall\mathbb{R}$-hard. van der Zander et al. need tree-shaped structure to get polynomial-time algorithms. These authors and Garcia et al. also give the useful notion of $k$-identifiability, meaning finitely many valid solutions rather than one. Registration objectives are non-polynomial and depend on the image data, so symbolic certification is out of reach. We borrow the vocabulary instead and estimate the number of solutions $k$ empirically, as the number of modes in the recovered estimates.

On the registration side, three papers motivate the gap. Towards Saner Deep Image Registration documents non-unique deep registration outputs: 1.522% folded voxels for VoxelMorph on IXI. Its authors explicitly leave multimodal "sanity" open, and our self-perturbation test is a multimodal analogue of their self-sanity check. R2N2 cuts B-spline parameter counts by 92% at almost the same error (1.16 vs 1.10 mm TRE). It also warns of "over-registration" by compensating Gaussian parameters. That suggests many registration parameters are redundant, but it never measures which ones are. CLAIM recovers a six-parameter camera-LiDAR transform using mutual information plus monocular-depth structure, but needs a rough initial guess. Our map would show which of the six parameters each cue actually pins down.

Research Program

The core estimator works as follows. Let $\hat\theta$ be the registrator's solution for a fixed image $F$ and moving image $M$. Draw perturbations $\delta\sim p(\delta)$ from a prior centred at zero, with independent coordinates. Warp $M$ by $\delta$, re-register from the same initialization, and record the new estimate $\hat\theta(\delta)$. The recovered offset is $r(\delta)=\hat\theta(\delta)-\hat\theta$. Per parameter $i$, and for parameter pairs $(i,j)$, we compute

$$S_i = \frac{I(\delta_i; r)}{H(\delta_i)}, \qquad C_{ij} = I(\delta_i;\delta_j \mid r),$$

using Gaussian kernel density estimates as in Capellari et al. In these formulas, $I$ is mutual information, $H$ is entropy, $S_i\in[0,1]$ is a normalized sensitivity, and $C_{ij}$ measures how much knowing the output couples two perturbations that were independent a priori. Separately, we count the number of modes $k_i$ in the conditional distribution of $r_i$ given $\delta$. A large $k_i$ flags discrete ambiguity, which neither $S_i$ nor $C_{ij}$ alone captures.

The map is defined by where the registrator behaves well. "Identifiable" means the registrator recovers a perturbation it was never told about. The map does not require knowing the true alignment, which is exactly what is unknown on clinical data.

We apply the estimator to three transformation families of increasing size. The first is six-parameter rigid camera-LiDAR extrinsics. The second is affine plus coarse B-spline grids, following Rueckert et al., for ultrasound-to-MRI and histology. For grids, the conditional-information computation runs only over spatially adjacent control-point pairs, which keeps the pairwise count linear in grid size. The third family is sparse Gaussian warps in the style of R2N2.

The intervention uses the map. Parameters with low $S_i$ get frozen or heavily regularized. Parameter pairs with high $C_{ij}$ get tied, for example by merging neighbouring control points or reparameterizing rotation-translation pairs. Discrete ambiguity triggers multi-start search with an explicit choice among modes. We then re-register under the reduced parameterization.

As a secondary use, the same sensitivity score $S_i$ compares similarity cues on equal terms. The cues are raw-intensity mutual information, wavelet-subband mutual information of the kind used as a loss in CWMI, and CLAIM's depth-structure term. This tests Gap 4 of the research file directly: whether richer cues restore identifiability without labels.

Baselines

The naive reference point is the unmodified classical registrator, run with its default uniform regularization. For volumes this is NiftyReg, and for camera-LiDAR it is CLAIM's search.

On ReMIND2Reg (intraoperative ultrasound to MRI), Learn2Reg 2024 reports NiftyReg at 2.87 mm TRE, ahead of the best deep entry at 4.42 mm. This is the strongest existing method on our hardest volumetric task. We will run NiftyReg with identical images, initialization, and a hyperparameter search budget equal to the one we give our reduced parameterization. Beating it means lower median TRE together with no increase in folded voxels, across at least three seeds.

On COMULISglobe second-harmonic-generation-to-bright-field microscopy, the winning classical entry, VROC, reports 2.64 pixels TRE (Learn2Reg 2024). We use it as an external reference.

On KITTI, CLAIM's multi-frame variant reports rotation error below 0.2°, translation error around 3–4 cm, and yaw error below 0.1° (CLAIM). For this task we evaluate two things. The first is whether the map correctly predicts which axes carry the residual error. The second is whether mode-aware multi-start removes CLAIM's dependence on initialization.

For the prediction task, the strongest competitor is the local curvature of the similarity objective. That means the inverse Hessian, or Fisher-information approximation, of mutual information at $\hat\theta$, which yields per-parameter variances and correlations at the cost of a single registration. No reported result found on Lacuna for Hessian-based identifiability or uncertainty maps in multimodal registration, so we will implement it on the same registrator. The second competitor is the empirical covariance of the same perturbation responses $r(\delta)$. This baseline is the key ablation. It uses exactly our data and compute, and differs only in replacing mutual-information quantities with second moments. The third competitor is a content heuristic: local gradient magnitude, or local joint-histogram entropy, as an error predictor.

The proposal beats these baselines if its error-prediction rank correlation and failure-detection AUROC are higher than those of both the Hessian and the covariance ablation. We also require the advantage to concentrate on cases with multiple solutions ($k_i>1$). If it does not, the information-theoretic ingredient is not what is doing the work.

Why not just…?

Why not just use the Hessian or Fisher information of the mutual-information objective? It is cheaper, and for well-behaved, near-quadratic optima it should agree with our map. Capellari et al. report that Fisher-based analysis struggles in strongly non-linear systems and cannot separate insensitivity from compensation. However, that finding comes from structural mechanics, not images. The real test is our head-to-head on ReMIND2Reg and KITTI. We expect the Hessian to be competitive on rigid camera-LiDAR and to lose on ultrasound, where estimated mutual-information landscapes are noisy and have many modes. If it matches us everywhere, we will report the Hessian as the practical tool.

Why not just measure the variance of the registration output under random perturbations or test-time augmentation? This is the most serious threat, because it uses the same expensive re-registrations we do. Per-parameter variance cannot detect compensation, but a full covariance can detect linear compensation. The only remaining gap is non-linear or multi-solution structure. That is why the covariance version is our main ablation, and why we stratify results by estimated $k_i$. An honest outcome could be that covariance suffices. In that case the contribution shrinks to the perturbation protocol itself, which is still a usable multimodal sanity check.

Why not just register segmentation labels, as Hu et al. did? Label supervision produced 3.6 mm versus over 24 mm for mutual information on prostate MRI-ultrasound (Hu et al.). But it needed expert annotation across 76 patients. It is also unavailable for unlabeled settings like LUMIR's more than 3,000 scans, or for microscopy. Labels also leave the question of why mutual information failed unanswered, and the sensitivity score answers it. Where labels do exist, we will check whether our map singles out the regions where label supervision helped most.

Why not just add stronger smoothness regularization? Uniform smoothing penalizes informative and uninformative parameters equally. Duan et al. show that smoothness penalties alone leave folding and self-inconsistency in place. Our NiftyReg comparison will sweep the global regularization weight over the same budget. If the best uniform weight matches map-guided reduction, the intervention claim fails.

Proposed Experiments

The first experiment is a controlled check of the estimator. We use synthetic multimodal pairs built by applying an invertible intensity remapping and a known warp to OASIS brain MRI, which gives exact ground truth. We plant a textureless region, a mirror-symmetric structure, and a rotation-translation trade-off. Success means $S_i$ falls in the textureless region, $C_{ij}$ rises for the planted trade-off, and the mode count $k_i$ equals 2 at the symmetry, all where the Hessian baseline's signal is weaker.

The second experiment is the decisive real-data test, on ReMIND2Reg and COMULISglobe using their landmarks. Per case and per region, we compute the identifiability map from about 100–200 re-registrations on a coarse grid. We report Spearman correlation with landmark TRE, and AUROC for flagging cases above 5 mm error. We compare against the Hessian, covariance, and content-heuristic baselines. The pre-stated falsification criterion is that our AUROC fails to exceed the covariance ablation's by a margin outside seed variability.

The third experiment tests the intervention: map-guided freezing and tying, then re-registration. We report TRE and folded-voxel percentage against tuned NiftyReg (2.87 mm) and VROC's reported 2.64 pixels.

The fourth experiment uses KITTI camera-LiDAR. We compute the six-parameter map separately for intensity mutual information, depth structure, and wavelet-subband mutual information. We test whether mode-aware multi-start recovers CLAIM-level accuracy (below 0.2°, around 3–4 cm) from initializations where CLAIM's search fails.

If SmartTarget data is accessible, a final diagnostic asks whether the over-24 mm failure of mutual information in Hu et al. shows up as global insensitivity, compensation, or many-mode ambiguity. That question has never been answered.

Compute is the main practical risk, since each case needs many registrations. We limit it with coarse grids, GPU NiftyReg, and adjacency-restricted pairs, and we report wall-clock cost next to every accuracy number.