Seeing What the Pyramid Throws Away: Loss-Triggered Wide Search for Coarse-to-Fine Motion

Most motion estimators work coarse-to-fine. They shrink both images until a large movement becomes a small one, estimate motion at that tiny scale, then pass the estimate up through progressively sharper images, correcting it a little at each level. The scheme has a known blind spot. A thin vessel, a small lesion or a small fast object can be blurred out completely by the time the image is shrunk. The coarse level then assigns it the background's motion, and every finer level only searches a few pixels around that wrong guess, so the error is never repaired. Two literatures describe this same failure from opposite sides. In optical flow, HMFlow shows small fast objects vanishing at the 1/64 pyramid level, with PWC-Net reaching a foreground end-point error of 87.01 on SFChairs. In gigapixel pathology, Thinking in Scales (PathCTM) names the "hard attention bottleneck": anything invisible at low resolution receives zero saliency and is pruned for good.

Our proposal starts from a simple observation. You do not need to run a network at full resolution to know where the coarse levels are blind. The detail that each shrinking step throws away can be computed exactly and cheaply, with one filtering pass per level of a Laplacian pyramid. That pass records, pixel by pixel, the image content that the next coarser level loses. We propose to use this measured loss, not any coarse-level judgment, to decide where a motion estimator should stop trusting the inherited coarse estimate. At those few locations it reopens a wide displacement search at native resolution, and the result of that search competes with the inherited estimate instead of being added to it as a small correction.

Core Claim

The failures of coarse-to-fine motion estimation on small structures are predictable before any motion is estimated. They occur where most of the local image energy lies in frequency bands finer than the level at which the displacement was first resolved. Spending a fixed, small budget of wide native-resolution matching at exactly those locations should recover most of the accuracy of a dense full-resolution search, at a cost close to that of a standard pyramid.

The mechanism is concrete. Let $x$ be an image and $\downarrow,\uparrow$ the pyramid's blur-and-subsample and interpolation operators. The detail band removed at level $l$ is $r_l = x_l - \uparrow\downarrow x_l$. We define a per-location visibility profile from the local energies $e_l(p)=\sum_{q\in N(p)} r_l(q)^2$ over a small neighborhood $N(p)$. We then define the blindness score at the level $L^\*$ where the large displacement is resolved:

$$b(p) = \frac{\sum_{l < L^\*} e_l(p)}{\sum_{l} e_l(p) + \epsilon},$$

where $\epsilon$ is a small constant that keeps flat regions from scoring high. A score near one means the structure at $p$ effectively does not exist at the level whose motion estimate it inherits. The top-$k$ locations by $b$, taken in both frames under a fixed budget, receive a native-resolution correlation search. Its radius is set by the largest coarse displacement in the surrounding window plus a margin. Each such location then chooses between the propagated estimate and the best wide-search match, using a learned gate that sees both matching costs. Smoothness is enforced only after that choice.

This differs from the three closest designs in specific ways. HMFlow adds a dense global-matching branch everywhere, and that branch is itself an encoder-decoder that downsamples, so faint structures can still be erased inside it. Our search is sparse, it runs at native resolution, and it is triggered by measured loss. F3RNet keeps a dense full-resolution stream and must crop to $144^3$ subvolumes to fit in memory. We keep full field of view and pay the high-resolution cost only where the analysis says it matters. PathCTM decides where to zoom from low-resolution evidence, which is exactly why it prunes faint targets. Our trigger is computed from the high-frequency residual itself, so it cannot be blind to what the coarse level missed.

Research Program

The work has three parts, and the first is a falsifiable analysis rather than a method. We take trained coarse-to-fine estimators, PWC-Net on SFChairs and a VoxelMorph++-style pyramid on Lung250M-4B, and ask whether $b(p)$ predicts per-location error. We compare it against two natural competitors: the coarse level's own uncertainty (cost-volume entropy or flow variance) and raw image gradient magnitude. We measure this as precision-recall of each score for retrieving the worst 5% of errors. If blindness does not beat coarse uncertainty, the premise is wrong and the method half of the project should be abandoned or redesigned around uncertainty.

The second part builds the module. It is a drop-in stage for two backbones: a 2D pyramid flow network and a 3D pyramid registration network in the style of CorrMLP, whose correlation windows top out at $7^3$. The module computes the Laplacian-band energies once per input pair and selects $k$ locations. It runs wide correlation on native-resolution features at those locations, and the cost grows as $k(2R+1)^d$ for search radius $R$ in $d$ dimensions. On lung CT, for example, $k=2\times10^4$ and $R=16$ voxels give roughly $7\times10^8$ feature dot products, which is feasible on one GPU. The module then gates between the inherited estimate and the new candidate. Training uses the backbone's usual loss, plus a loss on the gate supervised by which candidate had lower error, where ground truth exists: SFChairs flow, and Lung250M-4B keypoints.

The third part probes the regimes where the idea should and should not help. In lung CT, most thin vessels move with the surrounding parenchyma, so the inherited estimate may already be nearly right and the gains may be modest. To separate "the module recovers lost motion" from "lungs did not need it", we build a controlled stress test on real CT. We insert synthetic thin tubular structures whose displacement departs from the parenchymal field by a known amount, and vary their width relative to the coarsest pyramid level.

Baselines

The naive reference points are the identity transform, which leaves 16.25 mm TRE (target registration error, the distance between corresponding landmarks after registration) on Lung250M-4B, and the unmodified pyramid backbone. On SFChairs the unmodified backbone is PWC-Net, with a reported foreground AEE (average end-point error) of 87.01 in HMFlow. These show whether the module does anything at all.

On SFChairs the strongest learned competitor is HMFlow itself, with a reported foreground AEE of 44.34. We will reimplement it on the same PWC-Net backbone, training schedule and data. We report our module under two parameter budgets: one matched to HMFlow's global branch, and a smaller one, since the claim is sparsity. Beating it means lower foreground AEE at equal or lower FLOPs, with no loss in background AEE.

On Lung250M-4B the strongest method in the file is the classical optimizer deeds, at 1.53 mm TRE, against 2.26 mm for VoxelMorph++ and 2.73 mm for PointPWC-Net, all reported in Lung250M-4B. We will run deeds with its published settings on the same test split. We will add our module to the VoxelMorph++-style backbone with the same keypoint supervision the dataset paper found worth nearly 50%. A clear win is matching or beating 1.53 mm while keeping learned-model inference times. A weaker but still meaningful result is closing most of the gap from 2.26 mm at the same backbone cost.

Three ablations isolate the key ingredient. In the first, the same $k$ wide-search locations are chosen by coarse-level uncertainty instead of $b$. In the second, they are chosen uniformly at random under the same budget. In the third, wide search runs at every location, which is the dense upper bound. A fourth ablation replaces the competing gate with standard residual addition, to test whether letting a fine level overrule the coarse estimate matters. All variants share data, epochs, and a fixed hyperparameter search budget.

Why not just…?

Why not just keep a full-resolution stream, as F3RNet does? This works when memory allows, and the dense-search ablation is exactly that upper bound. But F3RNet already had to crop to $144^3$ to make it fit, and a breathing displacement of 16.25 mm leaves such crops more easily than it stays inside them. Our experiments report accuracy against compute for the dense variant and the sparse module. If sparse selection reaches, say, 90% of the dense gain at a few percent of the extra cost, that settles it. If it does not, the dense design wins and we will say so.

Why not just trigger the extra search on the coarse level's own uncertainty? This is the most serious threat, because it is simpler and already common. Its weakness is the one PathCTM documents: a structure that is invisible at coarse scale can produce a confident, smooth, wrong estimate rather than an uncertain one. Whether that happens often enough to matter is an empirical question. The first-stage error-prediction study and the uncertainty-selection ablation answer it directly. If uncertainty selects as well as blindness, the contribution reduces to "sparse wide search at fine scale", and we would reframe the paper around that.

Why not just add an HMFlow-style global matching branch? It is a proven fix in 2D. However, it is dense, and it downsamples internally, so it inherits a weaker version of the blind spot for faint, low-contrast structures. The width sweep in the synthetic-structure test should show it degrading earlier than the native-resolution sparse search as structures get thinner. If it does not, HMFlow's design is enough and our selection rule adds nothing.

Why not just use deeds or another classical optimizer? On lung CT it is currently the best reported method. Deep Learning in Medical Image Registration: Magic or Mirage? also shows that classical tools generalize better across scanners than learned registration. We therefore treat deeds as the bar to clear, not a straw man. We also evaluate across the multicentric sources inside Lung250M-4B as a mild out-of-distribution check, training on some sources and testing on held-out ones. A learned model that wins in-distribution but loses on held-out centers has not beaten it.

Proposed Experiments and Decisive Checks

The first decisive check is the error-prediction study: on both SFChairs and Lung250M-4B, $b(p)$ must retrieve the worst coarse-to-fine errors better than coarse uncertainty and gradient magnitude. The second is the stress test on CT with inserted structures. We plot TRE on the inserted structures against their width in coarsest-level voxels. We expect the unmodified pyramid to fail sharply below about one coarse voxel. We expect the module to stay flat until the structures fall below native resolution, since the residual band still carries their energy until then. The third is the head-to-head on Lung250M-4B TRE against deeds and VoxelMorph++, with the same split, keypoint supervision and wall-clock time reported for each, plus held-out-center results. The fourth is SFChairs foreground and background AEE against PWC-Net and our HMFlow reimplementation at matched FLOPs.

We have no reported Sintel or KITTI end-point numbers for the 2D baselines from the file. If we add those benchmarks to check that the module does not hurt ordinary scenes, all comparisons will come from our own reruns. The proposal succeeds if blindness beats uncertainty as a trigger, and if the sparse module approaches the dense upper bound at a fraction of its cost. It fails, informatively, if coarse uncertainty selects equally well or if lung motion leaves nothing for fine-scale search to recover.