Local features are small image descriptors that let a computer recognize the same physical point in two photos taken from different places or at different times. Every 3D reconstruction, mapping and camera-localization system depends on them, and they still fail at night, in rain, and on shiny or transparent surfaces. The usual fix is to train on more real photos. Real photos, though, cannot tell a network one fact that matters most: whether a given pixel should have a match at all. A point on a mirror, a glass pane, or a specular highlight looks different from every viewpoint, and a point hidden behind an object has no match anywhere. Real training pipelines estimate matches with Structure-from-Motion (reconstructing 3D geometry from many photos), which cannot recover these cases. So networks either learn to force bad matches or learn to avoid them only indirectly.
A procedural 3D simulator knows these facts exactly. We propose training a detector-descriptor on procedurally generated multi-view 3D scenes, using the renderer's exact depth, visibility and material information to supervise a per-pixel "correspondence validity" map. The network is told directly which pixels are occluded, specular or transparent and must not be matched. The claim to test is this: the simulator's most useful export for feature learning is not realistic images but labels of where correspondence is physically undefined. We further expect that this supervision removes a known trade-off, in which adding reflective materials to synthetic data helps on reflective scenes but hurts on ordinary ones.
Core Claim
Two pieces of prior work motivate the idea. What Makes Good Synthetic Training Data for Zero-Shot Stereo Matching? used the Infinigen procedural generator to vary scene factors for stereo training. It found that realistic room layouts with randomized floating objects, randomized camera baselines and non-photorealistic lighting transfer best to real data. It also found that adding glass and metal materials improves the reflective Booster benchmark by 25% but degrades the diffuse Middlebury benchmark. R2D2 showed that a feature network benefits from a learned "reliability" map that suppresses repeatable but ambiguous regions. However, R2D2 learns that map only indirectly, through a matching-ranking loss (average precision) on real web and Aachen images.
Our hypothesis is that the Booster/Middlebury trade-off comes from forcing a network to match across non-Lambertian surfaces, meaning surfaces whose appearance changes with viewing angle. If the reliability head is instead supervised with the renderer's exact validity labels, reflective data should teach the network where to abstain rather than corrupt its descriptors on diffuse surfaces. The feature-learning side supplies the mechanism: R2D2's reliability head. The synthetic-to-real side supplies the regime and the labels: Infinigen with the stereo study's randomization recipe, extended from rectified stereo pairs to general views with arbitrary relative camera pose (six degrees of freedom, "6-DoF").
This differs from SuperPoint in two ways. SuperPoint bootstraps from 2D synthetic shapes and then applies random 2D planar warps (homographies) to real images. It therefore never sees parallax (nearby objects shifting relative to distant ones as the camera moves), occlusion, or view-dependent reflectance, and it has no validity supervision. It also differs from the stereo study, which trains dense disparity along rectified epipolar lines and produces neither descriptors nor any abstention mechanism.
Research Program
We render multi-view image pairs of Infinigen indoor and outdoor scenes using the InfinigenStereo codebase generator. We keep its best-transferring factors (floating clutter and non-photorealistic lighting) and replace the fixed stereo baseline with random 6-DoF relative poses spanning small to wide baselines. The same geometry is re-rendered under independently sampled lighting, including very dark scenes with sparse point lights, so that day-to-night appearance change is present without real night data.
For each pixel $x$ in view $A$, the renderer gives its 3D point, its projection $x'$ in view $B$, and the surface material there. We define a validity label $v(x)=1$ only when three conditions hold: a depth test confirms the point is visible in $B$; the surface's specular and transmissive fractions are below a threshold; and the point is not on a depth discontinuity. Otherwise $v(x)=0$.
The network is the R2D2 architecture from the public naver/r2d2 code, chosen because its reliability head $R$ already exists. This lets us change the supervision and nothing else. The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{AP}}(X, R;\, x \leftrightarrow x') + \lambda\, \mathrm{BCE}\big(R(x),\, v(x)\big) + \mathcal{L}_{\text{rep}}(S), $$
where the three terms are as follows. $\mathcal{L}_{\text{AP}}$ is R2D2's average-precision loss on descriptors $X$, weighted by reliability $R$, computed over exact rendered correspondences. $\mathrm{BCE}$ is binary cross-entropy between predicted reliability and the renderer's validity label, and $\lambda$ sets its weight. $\mathcal{L}_{\text{rep}}$ is R2D2's repeatability loss on the detection map $S$.
A second stage tests whether a small amount of real data can be added without destroying synthetic robustness. Following Robust Synthetic-to-Real Transfer for Stereo Matching (DKT-Stereo), a frozen copy of the synthetic model supplies descriptor and reliability pseudo-labels. Real Structure-from-Motion correspondences are kept only where they agree with those pseudo-labels. That paper plays two roles here. It constrains the design, since it shows naive fine-tuning on Booster pushed ETH3D error to 93.31%. It also supplies the method, since its teacher-student filtering held that error to 2.60%. We transfer this filtering from dense rectified disparity to sparse 6-DoF descriptors.
Evaluation uses the file's benchmarks. On HPatches we report mean matching accuracy at 3 pixels and homography accuracy. On ScanNet and MegaDepth we report relative-pose AUC at 5°, 10° and 20°. On Aachen Day-Night we report the percentage of night queries localized at (0.5 m, 2°), (1 m, 5°) and (5 m, 10°). On 4Seasons we report high-precision map-based localization. Booster's ground-truth disparity also lets us score sparse matches separately on reflective/transparent pixels and diffuse pixels, which directly measures the trade-off.
Baselines
The naive reference is SIFT, which the 4Seasons benchmark reports at 11.5% high-precision localization. A second naive reference is SuperPoint with nearest-neighbour matching. SuperGlue reports that nearest-neighbour matching reaches 36.4% AUC@20° on ScanNet. These show how much any learned feature adds.
The strongest directly comparable competitor is released R2D2. The r2d2_WASF_N16.pt checkpoint reaches HPatches MMA@3px of 0.721 (naver/r2d2). R2D2 localizes 45.9% of Aachen night queries at (0.5 m, 2°), against 42.8% for SuperPoint (R2D2). It is the right opponent because it uses the same architecture trained on real data with implicit reliability. Any gain can therefore be attributed to data source and supervision rather than architecture. We use the released checkpoint and also retrain it with our exact schedule, iteration count and hyperparameter search budget. All methods share one matcher (mutual nearest neighbour) and the same localization pipeline, so the comparison isolates the features.
For context we also report the heavier matchers. LoFTR, which matches dense features with transformers and needs no detector, reports 65.9% homography AUC@3px on HPatches, 22.06% AUC@5° on ScanNet and 52.8% AUC@5° on MegaDepth. SuperGlue, which matches detected keypoints with a graph neural network, reports 51.8% AUC@20° on ScanNet. On 4Seasons, SuperPoint+SuperGlue reaches 40.3% (4Seasons). We do not expect a sparse descriptor with nearest-neighbour matching to beat LoFTR on MegaDepth. The meaningful comparison is Aachen night and 4Seasons, where appearance change dominates.
Two ablations isolate the key ingredient. The first trains the same network on the same synthetic renders with R2D2's implicit reliability only, setting $\lambda=0$. The second holds scene content fixed but generates pairs by homography-warping single rendered views, SuperPoint-style, instead of rendering true second views. Beating R2D2 means higher Aachen night localization at (0.5 m, 2°) than 45.9%, reproduced under our retraining, and higher 4Seasons high-precision localization than R2D2 under an identical pipeline. The validity-supervision claim requires the $\lambda>0$ model to beat the $\lambda=0$ model on Booster reflective-region match precision without losing precision on diffuse regions.
Why not just…?
Why not just train on real reconstructions like MegaDepth and ScanNet? This is the most serious threat. LoFTR and SuperGlue reach strong numbers this way, and real images carry no rendering artifacts. The weakness is the labels. Structure-from-Motion depth is missing or wrong exactly on glass, highlights and night imagery. Even real-trained pipelines reach only 40.3% on 4Seasons. We do not argue around this. We train the same architecture on MegaDepth under equal compute and compare. If real training matches us on Aachen night and 4Seasons, the synthetic-only claim fails. In that case the result reduces to whether renderer validity labels add value on top of real data, which the second-stage experiment tests.
Why not just freeze a foundation encoder such as DINOv2, as FAINT does? Synthetic vs. Real Training Data for Visual Navigation (FAINT) used a frozen, web-pretrained visual encoder to reach 90% real-world navigation success from simulation. But its output is a robot action, not a pixel-accurate correspondence. Such encoders emit features on a 14–16 pixel grid, and LoFTR's accuracy depends on refinement at half resolution. We still include a frozen-DINOv2 variant with a trained fine-resolution head, on the same synthetic data. It tests whether appearance invariance can simply be borrowed rather than learned.
Why not just add reflective materials and let R2D2's implicit reliability sort it out? If average-precision training already learns to suppress specular pixels, explicit labels are redundant. The $\lambda=0$ ablation answers this directly. If the $\lambda=0$ model shows no diffuse-region degradation on Booster or Middlebury pairs, then our mechanism is unnecessary. The honest conclusion would then be the narrower one: 6-DoF procedural data is good for features.
Why not just fine-tune on real target data? DKT-Stereo documents catastrophic loss of cross-domain accuracy from naive fine-tuning. We run naive fine-tuning on Aachen daytime Structure-from-Motion pairs as an explicit arm and report its effect on 4Seasons and ScanNet, alongside the filtered variant.
Decisive Checks
Three results decide the proposal. First, the 3D-versus-homography ablation on identical scene content must show that true parallax and occlusion improve ScanNet and MegaDepth pose AUC. Otherwise SuperPoint's planar shortcut is adequate. Second, on Booster, compare four models: with and without non-Lambertian materials, crossed with $\lambda=0$ and $\lambda>0$. For the claim to hold, the trade-off reported by the stereo study must appear at $\lambda=0$ and disappear at $\lambda>0$. Third, zero-shot on Aachen night and 4Seasons, the synthetic-only model must match or exceed R2D2 retrained on real data under the same pipeline.
A win on the second check alone would still be a useful finding: simulators should export validity labels, not just images. A loss on all three would indicate that the synthetic-to-real gap for local descriptors is not a label-quality problem.
References
- What Makes Good Synthetic Training Data for Zero-Shot Stereo Matching?
- R2D2: Reliable and Repeatable Detector and Descriptor
- SuperPoint: Self-Supervised Interest Point Detection and Description
- Robust Synthetic-to-Real Transfer for Stereo Matching
- 4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions
- LoFTR: Detector-Free Local Feature Matching With Transformers
- SuperGlue: Learning Feature Matching With Graph Neural Networks
- Synthetic vs. Real Training Data for Visual Navigation