Many vision pipelines receive images that are low-resolution, compressed or noisy, and they "fix" them with a restoration network before estimating camera motion. The geometry stage increasingly relies on affine correspondences. These are feature matches that carry not only a point location but also a small $2\times2$ matrix describing how the neighborhood around the point is locally stretched, rotated and sheared between the two views. Because each such match supplies six constraints instead of two, a robust estimator built on random sampling and consensus (RANSAC) needs only two matches per guess instead of four or five. That shrinks the number of random guesses it must try by one to two orders of magnitude.
The local affine matrix is a derivative, and derivatives are exactly what restoration networks corrupt. Restorers trained to minimize pixel error output the average of all plausible clean images, which is blurry and flattens the gradients that affine estimation needs. Restorers trained for realism produce sharp but partly invented texture, which yields confident affine matrices that describe hallucinated rather than physical surfaces. We propose to stop treating the restored image as the input to geometry. Instead, a stochastic restorer draws several plausible clean images, affine correspondences are extracted from each, and their spread across samples is used as a per-match uncertainty. That uncertainty drives filtering and early rejection of bad guesses inside RANSAC.
Core Claim
The perception-distortion tradeoff, proved by The Perception-Distortion Tradeoff, says that no single restored image is both faithful on average and realistic. The analogue for geometry is simpler than it looks. If $x$ is a clean image drawn from the restorer's posterior $p(x\mid y)$ given the degraded observation $y$, and $A(x)$ is the affine matrix a detector extracts at a matched feature, then the quantity geometry actually needs is
$$\bar A = \mathbb{E}_{x\sim p(x\mid y)}[A(x)], \qquad \Sigma_A = \mathrm{Cov}_{x\sim p(x\mid y)}[A(x)].$$
Here $\bar A$ is the best estimate of the local affine map under squared error, and $\Sigma_A$ is its uncertainty. Neither is the matrix extracted from the blurry mean image, $A(\mathbb{E}[x\mid y])$, nor the one extracted from a single realistic sample. Because $A$ is a nonlinear function of the image, averaging before extraction and averaging after extraction give different answers.
Our claim is that estimating $\bar A$ and $\Sigma_A$ from a handful of posterior samples does two things. First, it removes hallucinated matches before sampling: matches whose affine matrix is unstable across samples, or which appear in only some samples, are discarded. Second, it supplies a realistic covariance to the uncertainty-based early-rejection scheme of Making Affine Correspondences Work in Camera Geometry Computation.
Both effects act on the outlier ratio $\epsilon$ in the iteration count $N=\ln(1-p)/\ln(1-(1-\epsilon)^k)$, where $p$ is the desired confidence and $k$ is the number of matches per guess. With $k=2$, a modest drop in $\epsilon$ compounds: going from $\epsilon=0.9$ to $0.8$ cuts $N$ roughly fourfold. We expect fewer iterations and higher pose accuracy than any single operating point on the perception-distortion curve, and than matching on the degraded images directly.
Relation to Prior Work
The affine-RANSAC line supplies the method. Theory and Practice of Structure-From-Motion Using Affine Correspondences establishes the six-constraint counting and the two-correspondence relative pose solver. Robust Homography Estimation from Local Affine Maps builds a two-point homography estimator with an "affine consensus" check, which compares the hypothesis's analytic derivative against the measured affine map, plus a threshold-free acceptance test. Making Affine Correspondences Work in Camera Geometry Computation shows that naive affine solvers fail on real images because the matrices are noisy. It fixes this with three pieces: intensity-based refinement that yields an analytic covariance, propagation of that covariance into each hypothesis, and early rejection of hypotheses before full verification.
That covariance assumes independent Gaussian pixel noise on a real photograph. Restoration errors are spatially correlated and structured, so we replace it with the posterior covariance above. Learning Affine Correspondences by Integrating Geometric Constraints learns affine shapes with an epipolar-derived loss and is the strongest affine extractor in the file. Like the others, it is evaluated only on clean images.
The restoration line motivates the gap. The Double-Edged Sword: Perception and Uncertainty in Inverse Problems proves that more realistic restorers have higher error uncertainty, but measures only image statistics, never their effect on derived geometric quantities. A Feature Reuse Framework with Texture-Adaptive Aggregation for Reference-Based Super-Resolution finds that perceptual and adversarial losses hurt correspondence accuracy inside reference-based super-resolution. It responds by changing the training schedule, whereas we leave the restorer alone and change how geometry consumes it.
On The Classification-Distortion-Perception Tradeoff constrains us. Its information-loss result says restoration cannot beat the best estimator operating on the degraded signal itself. We accept this. Our method creates no information. It exposes uncertainty that practical, non-Bayes-optimal feature extractors cannot see. The result also makes "match on the degraded images directly" a mandatory baseline that could in principle win.
Research Program
We first build a measurement study that the file shows nobody has run. HPatches provides ground-truth homographies, so the true local affine map at every point is known exactly: it is the homography's Jacobian. On HPatches pairs we apply controlled degradations: 4× bicubic downsampling with noise, and strong JPEG compression. We then restore with a family of models spanning the perception-distortion curve. The family runs from a pixel-loss network, through interpolations, to adversarial models of the kind in Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network and EnhanceNet, and ends with a stochastic restorer that can draw multiple samples. For each operating point we measure where it sits on the curve (PSNR and a no-reference perceptual score) and the error of extracted affine matrices against the true Jacobian. We also measure the matching inlier ratio and the RANSAC iterations needed to reach 99% confidence. This yields the first empirical map from perception-distortion position to affine accuracy and to $\epsilon$.
Second, we build the posterior estimator. We draw $S$ samples, typically between 4 and 16, from the stochastic restorer. We detect and match in each sample with the same extractor, using affine-covariant SIFT refined by intensity matching and, separately, the learned extractor of Learning Affine Correspondences by Integrating Geometric Constraints. We then associate matches across samples by location. For each association we compute $\bar A$, $\Sigma_A$, and a persistence score: the fraction of samples in which the match appears.
Correspondences with low persistence or large covariance trace are filtered out. The survivors enter the two-correspondence solvers. $\Sigma_A$ replaces the Gaussian-noise covariance in the early-rejection step of the Barath et al. code, and point-based local optimization finishes the estimate as in that work.
Third, we test whether the posterior covariance is calibrated. We check whether normalized affine errors against ground truth follow the predicted spread. Calibration is what justifies using the covariance to reject hypotheses rather than only as a heuristic ranking.
One cost is explicit. Drawing $S$ restorations is far more expensive than the milliseconds RANSAC spends. We therefore claim gains in accuracy, robustness and iteration count, not in end-to-end wall-clock time. We will report restoration cost separately and test whether a cheap stochastic restorer with few sampling steps preserves the benefit.
Baselines
The naive reference point is to skip restoration entirely. Under this baseline, affine-covariant features are extracted on the upsampled degraded images, refined with the intensity-based matching and covariance of Making Affine Correspondences Work in Camera Geometry Computation, and passed to the same two-correspondence solver with early rejection. The information-loss result makes this the conceptually strongest competitor. It runs on identical degraded inputs with the same RANSAC confidence, thresholds and iteration cap. Beating it means higher homography AUC on degraded HPatches and higher pose AUC at 5° on degraded MegaDepth, with fewer mean iterations.
The second baseline is the best single restoration. For each degradation, we choose the operating point on the perception-distortion curve that maximizes downstream pose accuracy on a held-out split. We give it the same tuning budget we spend choosing $S$ and the filtering thresholds. If our method cannot beat the best-tuned single image, posterior sampling adds nothing beyond picking the right restorer.
The third baseline is the strongest existing pipeline. It uses the learned affine extractor plus affine-based GC-RANSAC from Learning Affine Correspondences by Integrating Geometric Constraints, applied to restored images. On clean MegaDepth that paper reports AUC 65.5 at 5°, against 60.4 for DKM and 62.6 for RoMA. On HPatches it reports 0.851 mean matching accuracy. We found no reported result on Lacuna for any affine pipeline on degraded or restored MegaDepth, HPatches or KITTI, so all degraded-setting numbers will be produced by us under shared settings. The clean-image 65.5 serves as an upper reference.
Two ablations isolate the key ingredient. The first keeps the posterior samples but uses only the per-sample Gaussian-noise covariance instead of the across-sample covariance. The second keeps the across-sample covariance but disables persistence filtering. Together they separate "better hypothesis rejection" from "fewer hallucinated matches."
Why not just…?
Why not just run a modern point matcher on the super-resolved images? Point solvers need four or five matches per guess. Robust Homography Estimation from Local Affine Maps reports that its two-point affine method recovers homographies on the Extreme Viewpoint Dataset where four-point RANSAC fails entirely. It also reports that the two-point method does better in 1,000 iterations than the four-point method in 10,000. Hallucinated texture also harms point matching, per A Feature Reuse Framework with Texture-Adaptive Aggregation for Reference-Based Super-Resolution. We nonetheless include RoMA- and DKM-style point matching with five-point RANSAC on restored MegaDepth, because dense matchers may tolerate hallucination better than local affine shapes do. If they match our accuracy, the affine part of the proposal is unnecessary for this regime.
Why not just restore with pixel loss so nothing is hallucinated? The pixel-loss output is the posterior mean, which The Perception-Distortion Tradeoff shows averages away high frequencies. The resulting gradients are weak and the affine matrices are ill-conditioned. This is an empirical claim, and the first-stage sweep tests it directly by including the pure pixel-loss restorer as the curve's endpoint. If its affine error against the true HPatches Jacobians is as low as that of our posterior mean $\bar A$, we drop the sampling machinery.
Why not just take the degraded image's own uncertainty, via intensity-based refinement or test-time augmentation? This is the most serious alternative, since it is cheap and respects the information-loss principle. The refinement covariance in Making Affine Correspondences Work in Camera Geometry Computation models sensor-like noise. It does not model the ambiguity of which texture lies beneath a blurred patch, and that ambiguity is what posterior samples express. The naive baseline and the first ablation settle the question. If neither our calibration test nor accuracy separates the posterior covariance from the refinement covariance on degraded inputs, the proposal reduces to applying existing machinery to blurry images, and we will report it that way.
Proposed Experiments
The decisive experiments run on three benchmarks. The first is degraded HPatches at 4× bicubic downsampling with noise and at JPEG quality 10, scored by affine-matrix error against ground-truth Jacobians, homography accuracy and iterations to 99% confidence. The second is degraded MegaDepth, scored by pose AUC at 5°, 10° and 20°. The third is degraded KITTI, scored by rotation and translation error and RANSAC iterations, using the one-correspondence planar-motion solver of On Making SIFT Features Affine Covariant where it applies. On clean KITTI that paper reports runtime falling from about 80 ms to about 48 ms. We will also run the cross-domain SymB pairs as a stress test, since learned affine extractors already degrade there according to Robust Homography Estimation from Local Affine Maps.
Three results would confirm the proposal:
- Our estimate $\bar A$ has lower error than the affine matrix from any single restored image anywhere on the perception-distortion curve.
- Persistence filtering measurably lowers $\epsilon$, and iterations fall by roughly the factor the RANSAC formula predicts.
- Pose AUC exceeds both the no-restoration baseline and the best tuned single restoration.
The proposal would be refuted if the no-restoration baseline matches us on accuracy. That would mean the restorer contributes no usable geometry, consistent with the information-loss principle. It would also be refuted if the across-sample covariance is no better calibrated than the Gaussian-noise covariance.
References
- The Perception-Distortion Tradeoff
- Making Affine Correspondences Work in Camera Geometry Computation
- Theory and Practice of Structure-From-Motion Using Affine Correspondences
- Robust Homography Estimation from Local Affine Maps
- Learning Affine Correspondences by Integrating Geometric Constraints
- The Double-Edged Sword: Perception and Uncertainty in Inverse Problems
- A feature reuse framework with texture-adaptive aggregation for reference-based super-resolution
- On The Classification-Distortion-Perception Tradeoff
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
- EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
- On Making SIFT Features Affine Covariant