Displacement-Searched Affinity Priors for Change Detection Between Unregistered Images From Different Sensors

Suppose you want to know what a flood destroyed, and all you have is an optical satellite photo from before the event and a radar image from after it. The two images do not line up pixel for pixel, their pixel values mean physically different things, and you have no labeled examples of change for this sensor pair. Each of these problems has a partial fix in the literature, but each fix assumes the other two problems are absent.

Cross-sensor methods compare how each pixel relates to its neighbors, a pattern that stays similar across sensors, and they assume perfect alignment. Alignment-robust change detectors learn to warp one image onto the other, and they assume both images come from the same camera type and that plenty of training pairs exist. We propose to merge the two ideas. The same neighborhood-similarity pattern that makes cross-sensor comparison possible can also serve as a matching cost for registration. Searching over local displacements before comparing these patterns yields two outputs from one computation: a dense alignment field and a change map. Changed pixels are excluded from the alignment, and each alignment estimate carries its own confidence. The method needs no training data beyond the single image pair being analyzed.

Core Claim

The affinity-based change prior of Deep Image Translation With an Affinity-Based Change Prior (Luppino et al.) builds, for every pixel, a small matrix of how similar each pixel in its surrounding patch is to every other. It builds this matrix separately in each image. Because these relational patterns are roughly sensor-independent, the difference between the two matrices signals change even between radar and optical data. The file notes the method's weakness: under any spatial offset, the two patches cover different ground, so the difference spikes along every edge and marks static structures as changed. Our claim is that this failure can be fixed inside the prior itself. Instead of comparing the matrix at pixel $x$ in image $X$ with the one at the same pixel in image $Y$, we compare it against matrices at displaced locations $x+d$ and keep the best match.

Concretely, let $A^X_x$ and $A^Y_{x}$ be the patch affinity matrices at pixel $x$, defined as in Luppino et al. by Gaussian kernels on within-patch pixel distances, each image using its own bandwidth. Define the residual

$$r(x,d) = \frac{1}{k^2}\,\big\| A^X_x - A^Y_{x+d} \big\|_1 ,$$

where $k$ is the patch side length and $d$ ranges over a search window $D$. We then estimate a smooth displacement field $d(\cdot)$ and a change prior $\alpha(\cdot)\in[0,1]$ jointly by alternating on

$$\min_{d}\; \sum_x \big(1-\alpha(x)\big)\,c(x)\, r\big(x,d(x)\big) \;+\; \lambda \sum_x \|\nabla d(x)\|^2, \qquad \alpha(x) = \sigma\!\left(\frac{r(x,d(x)) - \tau}{s}\right).$$

The terms are defined as follows. The confidence $c(x)$ measures how sharply the residual curve over $D$ is minimized; it is the ratio of the second-best to the best minimum after excluding a neighborhood. The constant $\lambda$ sets smoothness. The squashing function $\sigma$ is a logistic function with threshold $\tau$ and scale $s$, both set from the residual distribution by Otsu-style thresholding.

The factor $(1-\alpha)$ keeps changed pixels from pulling the alignment, which is the mutual-guidance idea of E2ECDNet done without any learned network. The factor $c(x)$ gives each flow estimate its own confidence, so flat water or repetitive fields fall back to the smoothness term instead of drifting. The file lists this missing per-pixel flow confidence as an open problem for joint registration networks. The final warped pair and $\alpha$ then drive Luppino's X-Net translation and change stage unchanged.

What is new is not affinity comparison, and not joint registration plus change detection. It is using the cross-sensor invariant as the registration cost, so that one unsupervised, single-pair procedure handles misalignment and sensor heterogeneity together. General self-similarity descriptors for multimodal registration exist in the broader registration literature, though none appear in the retrieved file. We treat "register with such a descriptor, then run X-Net" as the main competitor, not as prior art to ignore.

Where This Sits Relative to Prior Work

Luppino et al. supply the method we build on and also motivate it: they show affinity differencing works across sensors, reporting Kappa 0.767 on the Texas pair and overall accuracy 0.984 on the China pair, but only on co-registered data.

Three alignment-robust detectors constrain the design space. DiffRegCD couples dense flow and change prediction using frozen diffusion features, the internal features of a pretrained image generator that denoises noise into images. It holds 72.0 F1 on perturbed VL-CMU-CD where the decoupled ChangeRD falls to 58.9. However, it is RGB-only, needs large synthetic training sets, and its authors name cross-sensor pairs as future work. Morphing Through Time bridges appearance gaps by generating intermediate frames. That route has no obvious meaning between radar backscatter and reflectance, and it costs about 2.7 s per pair. MatchCD and Compress-Align-Detect rely on homographies, which assume a flat scene and cannot model terrain parallax. MatchCD's matcher also depends on optical appearance features. We differ from all four by requiring no training corpus in either modality and by estimating dense, non-planar displacement.

AdaptDiff shows GAN translators can erase or invent thin structures across modalities. This is a warning against "translate first, then register," since hallucinated edges would become false anchors.

The longitudinal MS study Self-supervised Lesion Change Detection motivates the problem from the small-sample side. Its lesion false-positive rate often exceeds 0.80, which the authors attribute to residual registration error. That is the failure mode we target, though their setting is single-modality and is only a possible later extension here.

The small-sample constraint is intrinsic rather than imposed. Heterogeneous change detection benchmarks are individual scene pairs with no training split, so any method must work from one pair.

Research Program

We will first build the residual volume $r(x,d)$ efficiently. Naively the cost grows with the number of pixels, the number of displacements, and the fourth power of patch size. We will run it coarse-to-fine on an image pyramid, using small displacement windows at each level and random pixel subsets inside each patch, and we will measure how much accuracy these approximations cost.

Second, we will implement the alternating estimation. The flow is updated by discrete search followed by smoothing, weighted by $(1-\alpha)c$, and $\alpha$ is then updated from the new residuals. We will run a fixed small number of rounds and report convergence behavior.

Third, we will warp $Y$ by the final field and hand the pair and prior to X-Net and ACE-Net exactly as in Luppino et al. Any gain can then be attributed to alignment and to the prior rather than to a new translation network.

Fourth, we will study where the joint formulation matters. Our hypothesis is that it helps most when changed regions are large or cluster along structural edges, as with floods or razed districts. In those cases an unmasked registration would warp unchanged ground to fit changed ground, which is the chicken-and-egg failure described for decoupled pipelines.

Baselines

The naive reference is X-Net applied directly to the misaligned pair, as published. It shows how much misalignment costs and should degrade with offset size. Its aligned-data counterpart, X-Net on the original co-registered pair, is the ceiling. Its reported figures are Kappa 0.767 on Texas and OA 0.984 on China, from Luppino et al. For California, no reported result was found on Lacuna, so we will rerun it ourselves.

The strongest competitor is a two-stage pipeline that uses the same affinity residual for registration but without change masking or confidence weighting ($\alpha\equiv 0$, $c\equiv 1$), followed by X-Net. It shares every component except the coupling, so it isolates the key ingredient. We will also run a second two-stage variant restricted to a global homography estimated from the same residual. This mirrors the planar assumption of MatchCD and Compress-Align-Detect and isolates the value of dense displacement.

We will include a translate-then-register baseline. X-Net or ACE-Net first maps $Y$ into $X$'s appearance on the misaligned pair, then the RoMa dense matcher used in Morphing Through Time registers the translated image, and X-Net is rerun. Where code is available, we will also transfer DiffRegCD, trained on synthetic WHU-CD perturbations, onto the translated pair. We found no reported result on Lacuna for any of these on heterogeneous pairs.

All methods receive the same single pair, the same patch size and search radius where applicable, and the same hyperparameter budget: a fixed grid tuned on one held-out perturbation seed, then frozen across seeds and datasets. Beating the strongest competitor means higher Kappa than the unmasked dense two-stage pipeline at medium and large offsets, with non-overlapping seed intervals. It also means staying within 0.05 Kappa of the aligned X-Net ceiling at offsets up to about 10 pixels, while losing no more than 0.02 Kappa to X-Net at zero offset. Ablations will remove $\alpha$-masking, remove $c$, replace dense flow with a homography, and vary the search radius.

Why not just…?

Why not just register with a modality-independent similarity measure first and then run X-Net? This is the real threat. Our method contains it as the $\alpha\equiv 0$ ablation. The argument against it is the chicken-and-egg failure described for E2ECDNet and MatchCD: changed pixels have no true correspondence, and unmasked registration drags nearby unchanged ground toward them. We will measure flow error on unchanged pixels bordering change, split by changed-area fraction. If the two-stage pipeline matches ours even on the high-change California flood pair, we will report that coupling is unnecessary for single-pair heterogeneous detection. That would be a useful negative result, and it would redirect the contribution to the confidence term alone.

Why not just translate one image into the other sensor's appearance and use an off-the-shelf unimodal matcher? Translation networks such as X-Net are trained with losses that assume alignment, so on a misaligned pair they learn a blurred or shifted mapping. AdaptDiff also documents GAN translators inventing and erasing thin structures, which would become false matching anchors. The translate-then-RoMa baseline tests this directly. We expect it to do reasonably at small offsets and to fail as offsets grow, because translation quality and alignment depend on each other.

Why not just fine-tune a trained joint model like DiffRegCD? There is nothing to fine-tune on. Each heterogeneous benchmark is one unlabeled pair, and DiffRegCD's diffusion features come from RGB pretraining. Its own authors leave cross-sensor pairs open. The transferred-DiffRegCD baseline shows how far the RGB features carry once one side is translated.

Proposed Experiments and Decisive Checks

We use the Texas (Landsat 5 vs. EO-1 ALI), China (RADARSAT-2 SAR vs. optical) and California (Landsat 8 vs. Sentinel-1A SAR) pairs from Luppino et al. We apply controlled misalignment in three families: global affine shifts and rotations; smooth non-rigid fields drawn from random low-frequency displacement maps, which mimic parallax; and mixtures of the two. Magnitudes range from 0 to 20 pixels, with five seeds each.

We report Kappa, overall accuracy, F1 and area under the ROC curve for change. We report endpoint error of the recovered field on unchanged pixels, with the ground truth being the inverse of the applied warp. We also check calibration: whether low-$c$ pixels actually carry larger flow error.

Because the file identifies a gap between synthetic perturbations and real misalignment, we will add a realism check. We will take one available pair whose native registration is imperfect, compare changed-edge false positives before and after correction against the provided change map, and state plainly that this check is qualitative.

The decisive result has two parts. First, the gap between ours and the unmasked two-stage pipeline should grow with changed-area fraction and offset size. Second, the confidence term should cut flow error in flat and repetitive regions without hurting elsewhere. If neither holds, the displacement-searched prior reduces to known multimodal registration, and we will say so.