Learned visual odometry (VO), which estimates a camera's motion by tracking small image patches from frame to frame, is now trained almost entirely on synthetic video. In those renders every camera is a perfect pinhole: each scene point lands as a sharp dot. Real robot cameras use small, cheap, often wide-angle lenses. Near the image edges and away from the focus distance, these lenses smear points into asymmetric, color-dependent blobs. This proposal argues that such lens defects are not just "blur" that a robust network can shrug off. A comet-shaped smear (coma) or a color-dependent shift (lateral chromatic aberration) moves the apparent center of a patch by an amount that depends on where the patch sits in the frame and how far away the surface is. For a tracker, that is a systematic geometric bias, and it falls hardest on the peripheral patches that matter most for estimating rotation.
We propose to re-render a synthetic VO training set through real lens designs, with one physically consistent lens per sequence and sensor noise added after the optics. We then retrain a patch-based VO system on it. The test is whether this closes a measurable part of the gap between lenses on real hardware. The key point is that a real camera has the same lens for the whole trajectory, and its defects are tied to image position and depth. Per-frame random blur, which is how robustness augmentation is usually done, cannot represent that structure.
Core Claim
Optical aberrations harm learned VO mainly through field- and depth-dependent shifts in where patches appear to be. Plain loss of sharpness is the smaller effect. Training with temporally fixed, spatially variant, depth-coupled lens rendering, with physically ordered sensor noise, should reduce trajectory error and tracking failures on aberrated cameras. Space-invariant optical augmentation of equal strength and generic blur-plus-noise randomization should not achieve the same reduction.
Each part of the setup comes from prior work, and each prior work leaves a different piece open. Examining the Impact of Optical Aberrations to Image Classification and Object Detection Models ray-traces 100 patent lens prescriptions into field-dependent blur and shows detection mAP on MSCOCO falling from 46.8 to 25.7. This motivates our work and supplies our lens bank. However, it studies single static images, uses a guessed "virtual pixel size," and explicitly omits sensor noise and the image signal processor. Classification robustness to common optical aberrations introduces OpticsAugment, which convolves each training image with a random Zernike-polynomial kernel. Zernike polynomials are a standard basis for describing lens wavefront errors. The kernel in that work is uniform across the frame. OpticsAugment is our strongest augmentation baseline, not our method. Aberration-Aware Depth-From-Focus is the closest prior work in spirit. It shows that off-axis aberrations bias geometric estimates: depth-from-focus error drops from 0.4706 to 0.2095 MAE on simulated stacks when training uses a learned surrogate of the point spread function (PSF, the image a single point of light makes through the lens). It also transfers zero-shot to a real Canon lens. That work covers one lens, static scenes, and focus bracketing. We carry the idea to moving cameras, many lenses, and frame-to-frame tracking.
On the robotics side, Deep Patch Visual Odometry (DPVO) trains a sparse patch tracker with differentiable bundle adjustment purely on TartanAir and transfers to EuRoC, TUM-RGBD and ETH-3D. It is the system we modify. Its training data is pinhole-only. Deep Drone Acrobatics motivates the stakes. It shows that feature-track abstraction makes control policies transfer (100% vs 0% success on new backgrounds), but its authors note that the policy fails once the tracking frontend degrades. Our work targets exactly that frontend.
Research Program
The renderer takes TartanAir's sharp RGB frames, per-pixel depth, and known intrinsics. Known intrinsics remove the virtual-pixel guess that limits LensCorruptions, because sensor size and field of view are fixed by the simulated camera. For each of roughly 80 training lens prescriptions, we ray-trace PSF banks over radial field height, object depth and three wavelength bands. We then scale each lens to the TartanAir field of view. The lenses are assumed rotationally symmetric, so the PSF at any pixel is the bank entry at its radius rotated to its azimuth. Each frame is split into depth layers and convolved patchwise with the interpolated PSFs, per color channel. Yang et al.'s neural surrogate is an alternative if the banks prove too coarse. Twenty prescriptions, chosen to span wide-angle and compact designs, are held out entirely.
After the optics, we mosaic to a Bayer pattern, add signal-dependent shot noise and read noise, and apply a simple processing chain: demosaic, gamma, mild sharpening. This order matters. Sharpening amplifies coma tails, and demosaicing misreads color fringes as edges. Augmenting a clean image with noise and blur in arbitrary order represents neither effect. Each training sequence samples one lens, one focus distance and one noise level, all held fixed for the whole sequence.
Alongside retraining, we will run a diagnostic. On held-out lenses we measure DPVO's per-patch flow error as a function of field height and depth. We then decompose that error into a deterministic bias, the mean displacement, and a variance. The core claim predicts that bias dominates at the periphery. If variance dominates instead, generic noise robustness would suffice, and the mechanism we propose would be wrong.
Baselines
The naive reference is the released DPVO trained on pinhole TartanAir, evaluated unchanged. The file reports DPVO's TartanAir test error only relatively, as "40% lower error than DROID-SLAM" (Deep Patch Visual Odometry). For EuRoC, TUM-RGBD and ETH-3D, no reported absolute ATE was found on Lacuna, so we will re-measure all of these ourselves under one protocol.
The generic sim-to-real baseline retrains DPVO with Gaussian and disk blur plus additive Gaussian and Poisson noise, randomized per frame. This is the standard domain-randomization recipe. James et al. found that kind of recipe insufficient for grasping: 33–37% zero-shot success, against 70% for their canonicalization (Sim-To-Real via Sim-To-Sim). It remains the default a roboticist would try first.
The strongest existing method is OpticsAugment ported to VO, using space-invariant Zernike kernels sampled per frame. In classification it reports +21.7 points on OpticsBench for ResNeXt50 and +6.8 on ImageNet-C (Classification robustness to common optical aberrations). The detection follow-up reports an average 18% OpticsBench gain (Examining the Impact of Optical Aberrations). It is the only established optics-aware augmentation, so beating it is the real bar.
Two ablations isolate our ingredients. The first uses the same lens bank but resamples the lens every frame, which removes temporal consistency. The second uses the same lens per sequence but collapses each lens to its on-axis PSF, which removes field and depth dependence. A third variant puts noise before the optics, which tests the ordering.
All variants use the same DPVO architecture, the same number of training steps, the same random seeds, and severity matched by MTF50. MTF50 is the spatial frequency at which contrast falls to half. Müller et al. use it to equate kernel strength, so no variant wins simply by seeing harsher corruption. Beating a baseline means lower ATE RMSE and fewer tracking failures on held-out lenses and real captures, across three seeds. In addition, pinhole-domain error must stay within seed noise of the naive model.
Why not just…?
Why not just use Gaussian blur and noise randomization? Müller et al. show that isotropic blur badly predicts behavior under asymmetric aberrations: ImageNet accuracy falls from 76.6% to 42.7% on OpticsBench. However, VO is not classification, and the evidence may not transfer. The generic-randomization baseline at matched MTF50 settles this directly. If it matches our method on held-out lenses and the real rig, the bias-dominance claim is false and we will report that.
Why not just apply OpticsAugment's space-invariant kernels? A uniform kernel blurs every patch the same way. That cannot teach the tracker that apparent displacement depends on radius and depth, and so cannot teach it to discount or correct peripheral patches. This is the most serious threat, because the network might learn sufficient invariance anyway. The on-axis-only ablation and the field-height diagnostic decide the question: our claim requires that the gap appear specifically in peripheral-patch error.
Why not just track only central patches? This is a cheap and honest alternative. If the periphery is bad, restrict DPVO's patch sampling to an inner disk. It discards the wide-baseline geometry that constrains rotation, which should hurt most under aggressive rotations. We will run it with the radius tuned on validation lenses. If it matches our method, the practical conclusion is "crop," and the proposal's value shrinks to the diagnostic.
Why not just deblur before tracking? Spatially variant, depth-dependent inversion is ill-posed and amplifies noise. A restoration network also adds latency, and DPVO's 60–120 FPS budget leaves little room for it. We will still include a calibrated non-blind spatially variant deconvolution using the true lens PSFs. This is an upper bound for restoration that no deployed system would actually have, and we will report its accuracy and runtime.
Proposed Experiments
The synthetic test re-renders the TartanAir test split through the 20 held-out lenses at three focus settings. We report ATE RMSE, failure rate, and field-height-stratified flow error.
The zero-shot real test runs on EuRoC, TUM-RGBD and ETH-3D. These cameras are comparatively benign, so we expect small effects there. Their main role is to confirm that we do no harm.
The decisive check is a capture we will collect ourselves. A single rig with motion-capture ground truth records repeated hand-held and pan-tilt trajectories through the same textured scenes. The rig is fitted in turn with a well-corrected lens and with a low-cost wide-angle board lens, each at in-focus and deliberately mis-focused settings. The prediction is specific: the naive and OpticsAugment models show an ATE gap between the cheap and good lenses, and our model shrinks it substantially. The ablations should attribute most of that reduction to field- and depth-dependent rendering rather than to temporal consistency alone.
If the cheap-lens gap does not exist for the naive model in the first place, the premise fails, and that negative result is itself worth publishing for the sim-to-real community.
References
- Examining the Impact of Optical Aberrations to Image Classification and Object Detection Models
- Classification robustness to common optical aberrations
- Aberration-Aware Depth-From-Focus
- Deep Patch Visual Odometry
- TartanAir: A Dataset to Push the Limits of Visual SLAM
- Deep Drone Acrobatics (Extended Abstract)
- Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks.