Modern learned visual odometry estimates a camera's motion from video by following a small set of image patches from frame to frame. It then solves jointly for camera poses and patch depths, a step called bundle adjustment. The leading sparse system, Deep Patch Visual Odometry (DPVO), follows only about 96 patches per frame and picks their locations uniformly at random. Its authors report that random placement beats classical keypoint detectors such as SIFT and ORB. We think that finding says more about the detectors than about selection itself. Detectors are built to score well on proxy metrics like corner strength and repeatability. Those metrics are known to predict downstream geometric accuracy poorly: learned patch descriptors that win patch-matching benchmarks lose in full reconstruction pipelines (Comparative Evaluation of Hand-Crafted and Learned Local Features). We propose to learn a patch selector trained directly on the quantity that matters, which is how much a patch actually helps bundle adjustment. The score deliberately penalizes patches whose usefulness varies a lot across small perturbations.
In two sentences: run a random-patch odometry system many times, and record for every image location how strongly the solver trusts that patch and how much that trust fluctuates across runs. Then train a cheap head to predict "average trust minus fluctuation", and track the top-scoring locations instead of random ones. The expected payoff is either lower trajectory error at the same patch budget, or the same error with far fewer patches.
Core Claim
The claim is that a patch's value to odometry should be scored like a feature in stability selection, a family of statistical methods that keep only variables chosen consistently across resamplings of the data. A patch that looks good on average but whose contribution swings under lighting flicker, motion blur, or a different set of companion patches is a liability. Uniform sampling cannot avoid such patches, and cornerness detectors cannot recognize them. Following the Conservative Mean criterion of An integrated machine learning approach to stroke prediction, which ranked clinical features by mean minus standard deviation of performance across data splits, we define a location's utility as
$$ s(x) = \mu(x) - \lambda\,\sigma(x), $$
where $x$ is a candidate patch location, $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation of that patch's downstream contribution across perturbed runs, and $\lambda \ge 0$ trades stability against average utility. The contribution is measured as the bundle-adjustment confidence weight from DPVO's factor head, averaged over the patch's lifetime and discounted by its final reprojection residual.
The claim is specific and falsifiable. At fixed budget, the variance penalty ($\lambda > 0$) should matter beyond the mean alone. The gain should be largest where DPVO's random sampling is most exposed: low patch budgets, day-to-night or other appearance transitions, and scenes with moving or deforming content.
This differs from the closest work in the following ways. DPVO learns how much to trust patches it has already chosen, but never chooses where they go. Guided Feature Selection for Deep Visual Odometry reweights dense feature channels using only the previous frame's pose. It has no notion of variance across perturbations and no sparse budget. Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM selects frames rather than patches, using an embedding-distance criterion with no stability control. Impact of Facial Landmark Localization on Facial Expression Recognition motivates the proposal. It shows that inter-frame jitter, not static pixel error, is what corrupts downstream motion estimates, but it re-weights hand-picked facial regions rather than learning which regions to use.
Research Program
The first step builds teacher labels without new annotation. DPVO is run repeatedly on training sequences from TartanAir. Each run uses a fresh random patch draw, mild photometric jitter, and occasional dropped frames. This is a natural bootstrap: every location is covered many times, each time in a different context of companion patches and nuisance conditions. For each pixel region we accumulate the factor-head weights and residuals of the patches that landed there. This gives $\mu$ and $\sigma$ maps per frame. The procedure borrows the resampling logic of iterative Random Forests to discover predictive and stable high-order interactions, which kept only interactions recovered in over half of bootstrap replicates. It moves that logic from genomics tables to image locations.
The second step distills these maps into a small convolutional head on DPVO's existing feature encoder, so the expensive resampling happens only at training time. This addresses the known cost problem of stability selection. At inference the head scores a dense candidate grid. We then take the top $K$ locations under a spatial-spread constraint, a non-maximum suppression radius, because unconstrained selection clusters landmarks and harms triangulation. Appearance-based landmark selection for visual localization reports exactly this clustering failure. A small fraction of patches, roughly a sixth, stays random to preserve exploration of regions the head scores poorly.
The third step is analysis. We measure how well standard proxy scores (corner strength, descriptor repeatability, and the $\mu$-only score) correlate with per-sequence trajectory error, compared with the full $\mu - \lambda\sigma$ score. This directly tests the metric-misalignment hypothesis behind DPVO's "random beats detectors" result. We also inspect where the learned score is low. If it has quietly learned to avoid sky, moving vehicles, reflections, and surgical instruments, it is behaving as an implicit region-of-interest mask. In that case the key question becomes whether an explicit mask would do just as well. The Role of Pleura and Adipose in Lung Ultrasound AI shows explicit masking can remove shortcut regions (AUC 0.6553 to 0.7742), so the explicit mask is a serious competitor, not a strawman.
Baselines
The naive reference is unmodified DPVO with 96 uniformly random patches, run with its released weights and default settings. This is also the strongest existing method on this question. It is the system whose design choice we are challenging, and its authors report it reduces absolute trajectory error (ATE, the average distance between estimated and true camera positions) by 40% relative to DROID-SLAM on the TartanAir test split while running at 60 FPS on an RTX-3090 (Deep Patch Visual Odometry). No reported absolute ATE value for DPVO was found on Lacuna, so we will reproduce its numbers ourselves on identical splits before any comparison. Beating it means lower median ATE across seeds at 96 patches with no more than a few milliseconds of added latency, or matching its ATE with 48 patches or fewer.
Because our selector adds compute, the fairest competitor is random sampling at matched wall-clock time. That is DPVO with enough extra random patches, likely around 120 to 150, to consume the time our scoring head costs. If the selector does not beat this, the idea fails.
A classical-selection baseline places patches at maxima of image gradient magnitude, under the same spread constraint. This reproduces the "detector" condition DPVO compared against.
An explicit region-of-interest baseline applies an off-the-shelf semantic segmentation network to exclude sky, people, vehicles, and, on endoscopy, instruments, then samples randomly inside the remaining region.
Two ablations isolate our ingredients. The first uses the same head trained on $\mu$ only ($\lambda = 0$) and tests whether the variance penalty matters. The second is an oracle that selects patches using the true $\mu - \lambda\sigma$ maps computed on the test sequence itself, which bounds how much any selector could gain.
All learned variants use the same training sequences, the same DPVO backbone frozen or fine-tuned identically, the same number of training steps, and $\lambda$ and the spread radius tuned on a held-out validation split only. Every method is evaluated over at least five seeds, because random-patch systems vary run to run.
Why not just…?
Why not just track more random patches? This is the most serious threat. DPVO is fast enough that adding patches is cheap, and more random samples would dilute bad ones. The matched-wall-clock baseline settles it directly. We also sweep budgets from 32 to 192 patches for both random and learned selection. If the curves converge by 96, the selector only helps in a low-budget regime. We would then reframe the contribution as an efficiency result rather than an accuracy one, and report that honestly.
Why not just mask out dynamic and uninformative regions with a segmentation network? Explicit masking is well supported in medical imaging: Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease reports large data-efficiency gains from U-Net masking. It could plausibly capture most of the gain on driving and outdoor data. However, a semantic mask cannot see instability within static classes: repetitive facades, specular floors, textureless walls, or regions that become unreliable at night. It also carries the cascading-error problem of two-stage pipelines. The comparison against the mask baseline, split by sequence type, will show whether the learned score adds anything beyond "avoid known distractor classes". The key cases are the day-night transitions of TartanAir, where TartanAir: A Dataset to Push the Limits of Visual SLAM reports ORB-SLAM ATE exploding from 0.165 to 9.385, and the deforming tissue of the endoscopy data. We also test mask plus learned score together.
Why not just use DPVO's own confidence weights to prune patches after tracking starts? Post-hoc pruning discards patches only after their budget has been spent. It also uses a single noisy weight, which is exactly the $\mu$-only signal whose instability motivates this work (Marginal Contribution Feature Importance - an Axiomatic Approach for Explaining Data shows importance rankings scramble under redundancy). The $\mu$-only ablation, together with a variant that replaces the lowest-weight patches with fresh random ones mid-window, tests whether choosing locations up front beats reactive replacement.
Proposed Experiments
We train on TartanAir training environments and evaluate on the TartanAir test split, including its Hard and day-to-night sequences, where motion diversity is high. We add KITTI odometry, a lower-diversity driving setting where selection should matter less. We also evaluate on C3VD and StereoMIS endoscopy, where non-rigid tissue and instruments are the natural stress test. On those, EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting reports ATE of 0.23 mm on C3VD and 15.47 mm on StereoMIS as a reference point from a different, mapping-based system. We report ATE, tracking success rate, frames per second, and GPU memory, all on the same hardware.
The decisive checks are threefold. First, at 96 patches the learned selector must beat both default DPVO and matched-wall-clock random sampling on TartanAir Hard and on at least one endoscopy dataset. Second, the $\lambda > 0$ selector must beat the $\mu$-only selector, with the gap concentrated on appearance-transition and dynamic sequences; if the gap is zero everywhere, the stability component is not doing anything. Third, the learned selector at 48 patches should match random at 96. That would halve patch-graph cost, which is where the temporal-redundancy argument pays off.
In the correlation analysis, the full score should track per-sequence ATE more closely than corner strength or $\mu$ alone. That would indicate DPVO's advantage over detectors comes from escaping a misaligned proxy, not from randomness itself. If instead random-with-more-patches or the explicit mask matches us at equal cost, we will conclude that learned selection is not worth its complexity for sparse odometry. That result would still resolve the tension in DPVO's design claim.
References
- Deep Patch Visual Odometry
- Comparative Evaluation of Hand-Crafted and Learned Local Features
- An integrated machine learning approach to stroke prediction
- Guided Feature Selection for Deep Visual Odometry
- Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM
- Impact of Facial Landmark Localization on Facial Expression Recognition
- Iterative random forests to discover predictive and stable high-order interactions
- Appearance-based landmark selection for visual localization
- The Role of Pleura and Adipose in Lung Ultrasound AI
- Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease
- TartanAir: A Dataset to Push the Limits of Visual SLAM
- Marginal Contribution Feature Importance - an Axiomatic Approach for Explaining Data
- EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting