Global Motion, Local Kernels: Blur-Steered Features for Matching Under Camera Shake

When a camera moves during an exposure, each scene point is smeared into a short streak. Corners become line segments, and a local feature detector, which inspects only a few neighboring pixels, can no longer tell where along the streak the true point lies. It also cannot tell whether a smooth patch is a blank wall or a heavily blurred texture. The standard remedy for such "local receptive field" limits is to mix information across the whole image with attention or large multilayer perceptrons. For blurry keypoints, this remedy backfires. The most recent blur-aware detector reports that global mixing washes out the faint remaining gradients until keypoint heatmaps collapse, so it has to bolt a fixed local convolution back on to recover precision (SSMB).

We propose separating the two jobs. Global context should decide where each local kernel looks, not what the feature is. Blur caused by camera motion is low-dimensional. A handful of numbers describing how the camera rotated and translated during the exposure determine the direction and length of the smear at every pixel. We will train a small global head to predict this compact exposure-motion code from the whole frame. The code is then turned analytically into a spatially varying field of smear directions and lengths, and that field steers elongated local sampling kernels in a joint detector and descriptor. Features are localized and described in a coordinate frame aligned with their own blur. The detector also reports uncertainty that is honestly larger along the streak than across it. Global reasoning is squeezed through a bottleneck of about five numbers, so it cannot dilute local gradients. Meanwhile the local kernels become direction-aware in a way that fixed or axis-aligned kernels cannot.

Core Claim

Our central hypothesis is that, for camera-induced blur, a physically parameterized global motion code is a better use of long-range context than feature-level global mixing. We make three testable predictions. First, a detector-descriptor whose local kernels are steered by the predicted smear field will beat SSMB's fixed depthwise local branch on the hardest blur splits, and the gap will grow with streak length. Second, it will beat an unconstrained deformable kernel, where each pixel learns its own offsets. Third, it will produce blur-aligned keypoint uncertainty that improves relative pose estimation beyond what better match counts alone explain.

What is new relative to the closest work is the specific combination. We use a global, low-dimensional, physically interpretable motion estimate as the only channel through which non-local information reaches the local kernels. We apply it to a joint detector-descriptor rather than a restoration network or a detector alone.

Relation to Prior Work

Two blur-aware detectors motivate the problem and supply the training recipe. BALF detects keypoints directly on blurred images with a multi-axis gated MLP encoder supervised by SIFT keypoints from sharp images. SSMB removes the SIFT supervision and adds a local branch of fixed depthwise convolutions with channel attention to counter gradient dilution. Its training has two stages: synthetic-shape pretraining, then homographic and cross-blur consistency on paired sharp and blurred images, with a spatial diversity loss. We reuse SSMB's training recipe wholesale. We differ in the architecture: SSMB's local kernels are isotropic and grid-fixed, and its global mixing is axis-aligned. Both detectors also leave description to external descriptors trained on sharp patches, a gap we address directly.

Two dense restoration methods supply the physical insight but constrain the design. Gong et al. show that local blur kernels are ambiguous from local patches alone. They resolve the ambiguity with a whole-image network predicting a per-pixel motion field, followed by deconvolution taking 8.4 seconds per image. PAHS uses selective non-local attention because blur-induced displacements exceed convolutional receptive fields in video deblurring. Both confirm that blur needs non-local reasoning. Both apply it to dense restoration at non-real-time rates, and Nah et al. likewise report 3 to 7 seconds per frame. Neither produces geometric features. We keep their lesson and drop the restoration step. We also replace Gong's dense per-pixel prediction with a compact code, and we test that choice directly.

The receptive-field literature supplies the sampling mechanism. DAS shows that a lightweight deformable gate can sample far outside fixed kernel boundaries at linear cost. CCNet restricts attention to rows and columns for efficiency. Butterworth as Attention learns anisotropic, rotatable frequency filters. It notes that global spectral gating struggles when distortions vary across the image, which is exactly the case for rotational blur. Our kernels are deformable in DAS's sense, but their offsets are not free: they are generated from the physical smear field. The unconstrained DAS-style version is the key ablation.

Research Program

The exposure-motion code is a vector $\theta$ containing the camera's angular displacement over the exposure, $\omega \in \mathbb{R}^3$, plus two in-plane translation terms for residual shift. Given approximate intrinsics $K$, a pixel $x$ traces the path

$$x(t) = \pi\big(R(t\,\omega)\,K^{-1}\tilde{x}\big) + t\,\tau, \qquad t \in [-\tfrac{1}{2}, \tfrac{1}{2}].$$

Here $\tilde{x}$ is $x$ in homogeneous coordinates, $R(\cdot)$ is the rotation for a given axis-angle, $\pi$ projects back through $K$, and $\tau$ is the translation term. The tangent of this path gives a smear direction $u(x)$ and length $\ell(x)$ at every pixel. The field is curved and spatially varying even though $\theta$ has only five entries.

A global head predicts $\theta$ from a downsampled frame. This head is the only non-local pathway in the network. Each local kernel then samples features at positions $x + s\,\ell(x)\,u(x) + r\,n(x)$ on a small grid of $(s, r)$, where $n$ is the direction perpendicular to the smear, and combines them with learned weights. A bounded learned residual offset handles independently moving objects, which the camera-motion model cannot explain.

The detector head outputs a keypoint location and a 2×2 covariance whose principal axis is initialized to $u(x)$. The descriptor head pools features in the blur-aligned frame, so a blurred view and a sharp view of the same point are described in comparable coordinates.

Training follows SSMB's two stages and losses. We add one term: supervision of $\theta$ on sharp images blurred synthetically by averaging images warped along known rotations, which gives exact ground-truth motion codes. We also add a descriptor contrastive loss across sharp and blurred pairs from GOPRO. The whole system must stay within SSMB's reported throughput regime of roughly 33 FPS at VGA resolution on an RTX 3090.

Baselines

The naive reference point is a classical sharp-image pipeline, SIFT with RootSIFT normalization, run directly on the blurred images. The HPatches study found that such tuned handcrafted descriptors remain competitive. The skeptic's position, that nothing learned is needed, should therefore be measured rather than assumed.

The strongest existing method is SSMB, paired with HyNet descriptors as in its paper. It reports 40.1% mean matching accuracy at a 3-pixel threshold on the Tough split of Blur-HPatches, against 25.0% for BALF. It also reports 86.7% of nighttime Aachen Day-Night queries localized at 1.0 m/10°. BALF is a second baseline, with a reported 71.8% repeatability on Blur-HPatches Tough.

For fairness, we will retrain SSMB from its described recipe on the same GOPRO pairs, with the same synthetic pretraining, the same number of optimization steps, and the same hyperparameter search budget as our model. We report both our retrained number and the published one. Beating SSMB means exceeding its retrained mean matching accuracy on Tough by a margin larger than the seed-to-seed spread over three seeds, at matched latency. We expect the gain on Easy to be negligible; a gain there would suggest capacity rather than mechanism.

Three ablations isolate the key ingredient. The first replaces SSMB's depthwise local branch with an unconstrained DAS-style deformable gate of equal parameter count, testing whether physics-constrained steering matters or just flexible sampling. The second replaces the five-number code with a dense per-pixel flow head in the style of Gong et al., testing whether the global bottleneck itself prevents dilution. The third feeds the ground-truth $\theta$ at test time on synthetic data, giving an oracle ceiling for how much the motion estimate limits performance.

Why not just…?

Why not just drop a deformable attention module into SSMB? This is the most natural competitor. Free offsets can in principle learn to follow streaks, and DAS shows they help generic recognition. Our concern is that, under self-supervision on heavily degraded input, per-pixel offsets have little signal to learn from. This is the same regime in which SSMB observed heatmap collapse. The first ablation settles this directly. If the unconstrained gate matches our model at every severity, the physical parameterization is not earning its place, and we will report that.

Why not just deblur first and then detect? Published deblurring networks take seconds per frame (Nah et al., Gong et al.), and the blur-aware detector papers argue that restoration introduces ringing that misplaces keypoints. We will still run a deblur-then-SIFT and deblur-then-SSMB pipeline offline on Blur-HPatches. If it wins on accuracy, our claim narrows to the real-time regime.

Why not just fine-tune an off-the-shelf descriptor on blurred patches? This targets the descriptor gap cheaply. It cannot, however, change where the detector fires or give it streak-aligned uncertainty. We include SSMB plus a HyNet descriptor fine-tuned on the same GOPRO pairs as a baseline, so that any descriptor gain we claim is measured against equal blur exposure.

Why not just read the camera motion from a gyroscope? On robots this is a real threat. An inertial sensor measures $\omega$ directly and would make our global head unnecessary. The benchmarks here have no inertial data, so we test the question with the oracle-code ablation. If the image-predicted code performs within noise of the oracle, the image route costs nothing and works on sensorless imagery such as phone photos. If the oracle is far better, the lasting contribution is the steerable local kernel conditioned on any motion source, and we will say so.

Proposed Experiments

The primary benchmark is Blur-HPatches, split into Easy, Hard, and Tough. We report repeatability and mean matching accuracy at 1 to 5 pixel thresholds against the baselines above. Beyond the split averages, we bin every image pair by the ground-truth streak length and plot matching accuracy against it. Our hypothesis predicts a gap over SSMB that widens with streak length. A flat gap would indicate the benefit comes from something other than blur steering.

For downstream geometry, we evaluate relative pose on SSMB's ArchViz setting, where no reported result was found on Lacuna. We run it with and without our anisotropic covariances in the pose solver. This separates the value of better matches from the value of honest uncertainty. We also evaluate on the blur-augmented Aachen Day-Night protocol against SSMB's reported 86.7%.

To probe real blur that includes moving objects, we use held-out GOPRO pairs for matching and RealBlur images for qualitative inspection. We also check the residual-offset branch by ablating it on scenes containing independently moving objects.

Two diagnostic checks decide whether the mechanism works as claimed. The first measures heatmap entropy and the fraction of collapsed predictions for our model versus the global-mixing ablation, directly testing the dilution story. The second compares the predicted smear directions against ground truth on synthetic rotational blur, reported as angular error per pixel. If the smear field is badly wrong while accuracy still improves, the gain is not coming from physics, and the core claim fails. Latency is reported at VGA resolution against SSMB's roughly 33 FPS.