Many image-matching networks learn without human labels. Each training pair is made by taking one photo and bending it with a known random warp, such as a homography or a smooth elastic distortion. Because the warp is known, the network can be told exactly where every pixel went. To match pixels precisely, these networks often add skip connections, which are wires that carry high-resolution early features past the network's compressed middle stage to the layers that refine the match. The problem is that a synthetically warped image is not a natural photo. Resampling it leaves faint, warp-dependent traces: blur where the warp stretched the image, aliasing where it compressed it, and padding patterns at the borders. A high-resolution skip path is exactly the channel through which such traces can reach the matcher. The network may then learn to infer the warp from resampling artifacts instead of matching image content. That shortcut would vanish on real photo pairs and could quietly cancel the precision that skips are supposed to buy.
This note proposes a direct test of that hypothesis and a minimal fix. The test is a probe: from a single warped image, read through the skip features alone, can one predict the local stretch of the warp that produced it? If one can, the skip features carry information about how the image was made rather than what it shows. The fix is a pair-rendering protocol called artifact-matched rendering. Both views are produced by independent, anti-aliased resampling from a common higher-resolution source, with randomized resampling kernels, so neither view carries a warp-specific signature. The core claim is that high-resolution skips help self-supervised matching at strict pixel thresholds only once this leak is closed. Under naive single-sided warping, adding skips will improve accuracy on synthetic validation pairs but not on real pairs, or will even hurt there.
Where this sits in prior work
The two directions meet in a few papers, and none of them asks this question.
ResFPN adds multiple residual skips from high-resolution encoder stages into a feature pyramid for dense matching. It reduces the KITTI 2015 stereo outlier rate from 3.0% to 2.2% inside PSMNet. All of its training is fully supervised, however, and ground-truth flow on real or rendered scenes gives the network no warp artifacts to exploit. We reuse its skip design as the treatment and move it into the self-supervised setting it never tested.
Two papers supply the self-supervised setting and our baselines. DGC-Net trains a coarse-to-fine VGG-16 pyramid on synthetic affine, thin-plate-spline and homography warps, using only standard top-down connections. PDC-Net+ adds warp perturbations, composited moving objects and per-pixel uncertainty to the same warp recipe. Neither reports whether its refinement stages use image content or resampling cues.
Automatic Shortcut Removal motivates the hypothesis. It shows that self-supervised pretext models exploit low-level artifacts that humans fail to anticipate, and that an adversarial image-modifying "lens" can remove them. It also constrains our design. That lens works by suppressing high-frequency cues, which dense matching needs, so we treat it as a competitor rather than our method.
SEIS supplies a measurement tool. It scores how well a layer keeps spatial layout ("equivariance") versus category identity ("invariance") under transformations. It finds that U-Net skips restore equivariance lost at the bottleneck, but it studies only classification and segmentation. A high equivariance score cannot by itself tell real geometry from artifact geometry, and that ambiguity is exactly what our probe resolves.
Core Claim
Let $s(I')$ be the skip-path features of a warped image $I'$. Let $J(x)$ be the local Jacobian of the warp at pixel $x$, the 2×2 matrix describing how much the warp stretches and shears the neighborhood. We define the warp leakage of a skip path as the held-out $R^2$ of a small linear probe that predicts $J(x)$ from $s(I')$ at $x$, using no access to the source image $I$.
A model that matches content should carry little leakage, because the local stretch of an unseen warp is not visible in one natural-looking image. We predict three things:
- Under naive warping, leakage is high in skip features and much lower in bottleneck features.
- Leakage predicts the gap between accuracy on synthetic validation pairs and accuracy on real pairs.
- Artifact-matched rendering drives leakage toward a floor while preserving, and on real data improving, strict-threshold accuracy.
The floor is estimated with a control: the same probe applied to unwarped photos with randomly assigned Jacobians.
Research Program
We build one matching architecture in four variants. The two pyramid choices are DGC-Net's standard pyramid and the same pyramid with ResFPN residual skips ($h=2$). The two rendering choices are naive rendering and artifact-matched rendering. All four variants are trained self-supervised on identical source images, the PASCAL VOC and Tokyo Time Machine imagery used by DGC-Net, with the same warp families.
Artifact-matched rendering works as follows. We draw two independent warps from a source image downsampled at least 2× from its native resolution. Each view is resampled with an anti-aliasing filter drawn at random from a family of filters, and the border masks are matched. The relative warp between the two views supplies the supervision. Because both views are resampled, the "which view was warped" cue disappears, and because the kernel is randomized, the network cannot read stretch from blur shape.
The analysis has three layers. First, we measure warp leakage at every pyramid level, separately for skip and top-down features. Second, we run SEIS equivariance scores layer by layer, computed under real geometric changes (HPatches viewpoint pairs) and under synthetic warps. A layer whose equivariance is high only under synthetic warps is suspect. Third, we compute end-point error in a band of a few pixels around flow discontinuities and thin structures. That band is where fine detail matters and where smooth synthetic warps never supply training signal.
Baselines
The naive reference point is RootSIFT matching with ZCA whitening. The HPatches benchmark paper reports that this handcrafted descriptor is on par with or better than many learned ones. We run it with nearest-neighbor matching followed by homography fitting on the same HPatches pairs. It tells us whether any learned variant has earned its complexity at the 1-pixel threshold.
The strongest existing self-supervised dense matcher in the file at strict thresholds is DGC-Net. It reports 59% PCK at a 1-pixel threshold on HPatches Viewpoint V, against under 30% for the optical-flow baselines it compares with. We retrain DGC-Net in our pipeline with its published warp recipe and report both our reproduction and the cited 59%. Every variant is compared against our reproduction, with identical data, iterations, input resolution and hyperparameter-search budget. We use a fixed number of trials per variant on a held-out synthetic split, never on HPatches.
PDC-Net+ is the stronger general matcher. It reports up to 70% reduction in average end-point error on MegaDepth after discarding 30% of its least-confident matches. We could not find a reported absolute 1-pixel HPatches number for it on Lacuna. We therefore retrain its warp recipe on our architecture as a second rendering condition, since its perturbations and moving objects are a partial, untargeted answer to the artifact problem.
The key ablation is ResFPN skips with naive rendering. If this variant matches the artifact-matched skip variant on real data, the hypothesis is false.
Two shortcut-removal baselines sit on the same skip architecture. The first is heuristic augmentation (Gaussian blur, noise, JPEG compression, applied to both views). The second is the adversarial lens of Minderer et al., with its reconstruction penalty tuned within the same budget.
Beating the field means two results together. The artifact-matched skip model must exceed both our DGC-Net reproduction and the naive-skip model on HPatches PCK at 1 pixel, by more than the spread across three training seeds. It must also show lower boundary-band error on KITTI 2015 flow, all while measurably reducing leakage.
Why not just…?
Why not just apply blur and noise augmentation to both views? This is the most likely threat, because symmetric degradation partly hides which view was resampled. It does not remove the dependence of blur on the local stretch, which artifact-matched rendering targets with randomized kernels. Automatic Shortcut Removal found that hand-designed augmentations miss shortcuts that its learned lens found. The augmentation baseline settles this directly. If it reaches the same leakage and the same real-pair accuracy, we will report that the cheap fix suffices, which is still a useful finding for anyone training warp-supervised matchers.
Why not just use the adversarial lens? The lens removes whatever cues make the pretext task easy. In dense matching those cues include genuine edges and corners. Minderer et al. note that aggressive lens settings destroy useful spatial information, and they never applied the lens to sub-pixel matching. We expect it to lower leakage but also to lower 1-pixel accuracy. The boundary-band error measures that cost.
Why not just drop the skips and keep DGC-Net's standard pyramid? If skips mostly transmit artifacts, removing them is the simplest cure. The ResFPN results under supervision suggest that real detail is being left on the table: its boundary "bleeding" artifacts shrink when skips are added. Our 2×2 design measures whether that benefit survives self-supervision once the leak is closed.
Why not just train on rendered 3D data with true flow? Supervision from synthetic scenes avoids resampling artifacts, but it gives up the label-free setting and brings its own rendering gap. Novotný et al. (Probabilistic Introspection) note that planar warps cannot model out-of-plane motion, which is a limit we accept. Our question concerns warp-supervised training specifically.
Proposed Experiments
The decisive check comes first. We train all four variants and record two quantities for each: warp leakage at each skip level and the drop from synthetic-validation PCK at 1 pixel to HPatches Viewpoint V PCK at 1 pixel. The hypothesis predicts that naive-skip has the highest leakage and the largest drop, and that the artifact-matched variants close both. If naive-skip shows leakage near the unwarped-photo floor, the shortcut does not exist in practice and the proposal stops there with a negative result.
Next, we test generalization in two ways. We evaluate MegaDepth end-point error at matched coverage, both before and after confidence filtering, and KITTI 2015 flow error inside the discontinuity band. The band evaluation checks whether skips deliver detail where a smooth-warp prior fails.
Finally, the SEIS layer-wise audit tests whether the restored equivariance that SEIS observed in U-Nets reappears in the decoder of matching networks under real viewpoint change, or appears only under synthetic warps. The second outcome would mark a high equivariance score as an artifact rather than geometry.
References
- ResFPN: Residual Skip Connections in Multi-Resolution Feature Pyramid Networks for Accurate Dense Pixel Matching
- DGC-Net: Dense Geometric Correspondence Network
- PDC-Net+: Enhanced Probabilistic Dense Correspondence Network
- Automatic Shortcut Removal for Self-Supervised Representation Learning
- SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations
- HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors
- Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection