Errors That Patches Share: Non-Local Covariance for Learned Visual Odometry

Learned visual odometry estimates a camera's motion by tracking many small image patches and then solving for the pose that best explains where they moved. That final solve treats each patch's error as independent: a patch with a confident prediction gets more weight, but no patch's error is assumed to say anything about another's. This assumption fails in two predictable ways. Nearby patches are processed by convolutional filters that see overlapping pixels, so their errors move together. Distant patches are hit by the same scene-wide disturbances, such as motion blur, an exposure change, a repeated texture, or a moving object covering part of the view, so their errors move together too. When correlated errors are counted as independent evidence, the estimator becomes overconfident, and because the shared error does not average out, the pose estimate also picks up a bias.

We propose to predict the correlation between patch errors directly and to use it in the pose solve. The key observation is that the two sources of correlation call for different machinery. Short-range correlation follows from the receptive-field overlap of the frontend network and can be computed largely from its architecture. Long-range correlation comes from scene-level causes that a per-patch uncertainty head, built on local convolutions, cannot see at all. We therefore use a non-local module for a narrow purpose. It does not improve the features. It estimates a small number of shared error modes across the whole image. The resulting covariance, a diagonal term plus a local overlap term plus a low-rank term, can be inverted cheaply. Plugged into the patch-based pose solver, it should improve both trajectory accuracy and the honesty of the reported uncertainty.

Core Claim

The claim is that the size of a visual frontend's receptive field limits what its uncertainty estimates can represent. A per-patch uncertainty head with local support can learn how noisy each patch is. It cannot learn which patches will be wrong in the same direction, because the cause of that shared error lies outside its view. A simple calculation shows why this matters. For $n$ measurements with equal variance and pairwise error correlation $\rho$, the variance of their average is that of $n_{\text{eff}} = n / (1 + (n-1)\rho)$ independent ones; this is the standard variance of a mean of equicorrelated variables. With DPVO's 96 patches per frame and a modest $\rho = 0.1$, the solver believes it has 96 independent measurements when it effectively has about nine.

We model the inter-patch measurement error covariance for one frame pair as

$$\Sigma = D + K_{\ell} + L L^{T},$$

where $D$ is the diagonal of per-patch variances that existing systems already predict. $K_{\ell}$ is a sparse local term whose entry for patches $i,j$ decays with their image distance at a length scale initialized from the frontend's effective receptive field. $L$ is an $n \times r$ matrix with $r \ll n$, whose columns are shared error modes predicted by a global-context module. The pose solver whitens residuals with $\Sigma^{-1}$. Because $L L^{T}$ is low rank, the Woodbury identity and the matrix determinant lemma give the inverse and log-determinant at roughly the cost of handling $D + K_{\ell}$ plus an $r \times r$ solve.

Relation to Prior Work

The closest work is MAC-VO. It learns pixel-level flow uncertainty on a non-local FlowFormer backbone and converts it into full 3×3 metric covariances per keypoint, including correlations between the x, y and depth axes. Its correlations live within a keypoint, and different keypoints are treated as independent. Our proposal concerns correlations between patches, which is a different off-diagonal block. MAC-VO motivates this work, since it shows that richer covariance structure pays off in a pose-graph backend. It also supplies the backend weighting we reuse.

DPVO tracks 96 randomly sampled patches with per-patch confidence weights and reports 40% lower trajectory error than DROID-SLAM on TartanAir. It is both the host system and a constraint. Its success with sparse, spread-out patches suggests random spatial sampling already removes much of the short-range correlation. So our gain must come mostly from the long-range term, and the experiments test that directly.

Convolutional Neural Networks Regularized by Correlated Noise built activation-noise covariances from receptive-field distance and filter similarity. It used them only to inject noise during classification training, and only at the first layer. We borrow its idea that distance on the feature grid predicts correlation, and we use it for $K_{\ell}$. Unlike that paper, we estimate the correlation of prediction errors and use it inside an estimator.

Three papers supply the mechanism for the global modes. CCNet routes information across the whole image with two criss-cross attention passes, at 85% fewer FLOPs than a non-local block. ABFNet obtains a global receptive field through learnable Butterworth gating in the Fourier domain. It also argues that spatial attention acts as a low-pass filter, which is a concern here because blur-induced error modes are themselves smooth while texture-induced ones are not. Compact Generalized Non-local Network is the dense reference. We test criss-cross and spectral variants as the module that produces $L$.

Two lines of work from direction 1 constrain the design. Classical state augmentation, as in Jia et al. and Tang et al., models correlated noise explicitly but scales cubically with measurement dimension; the low-rank form is our answer to that cost. Covariance intersection, as in DInCIKF, stays consistent under unknown correlation by being conservative. It serves as the principled "don't model it, bound it" competitor.

α-MDF appears to contradict the premise. It replaces the Kalman gain with attention and reports large gains on KITTI without any explicit covariance. We take this seriously. An attention gain may learn to downweight redundant evidence implicitly. However, it reports no calibrated uncertainty, and calibrated uncertainty is what downstream fusion needs, for example with an inertial sensor. Our evaluation therefore scores calibration as a primary outcome alongside accuracy.

Research Program

The work starts with a measurement study, which can stop the project early. On TartanAir v2, ground-truth flow lets us compute the actual flow error of every tracked patch in DPVO and MAC-VO. We will estimate empirical error correlation as a function of image distance between patches. We will then ask whether it decays to zero, which would mean local correlation only, or to a nonzero floor, which would point to scene-level modes. We will also check whether frames with high floors coincide with blur, exposure change, dynamic objects or low texture.

Next we build the covariance head on DPVO. The diagonal term is DPVO's existing confidence. The local term $K_{\ell}$ is a distance kernel with two learned parameters per scale. The low-rank factor $L$ comes from a criss-cross attention block, or alternatively a spectral gating block, applied to the frontend feature map. Each patch's row of $L$ is read out at its location. Training uses the full-covariance Gaussian negative log-likelihood of flow residuals on TartanAir, computed with the determinant lemma, with the pose network frozen at first and then fine-tuned end to end through bundle adjustment. Rank $r$ is swept from 1 to 16.

As a secondary extension that we will pursue only if the spatial result is positive, we will test whether the global modes persist across consecutive frames. Persistence would make them temporally colored measurement noise. In that case a first-order autoregressive model on the $r$ mode coefficients, which is cheap because $r$ is small, would be the natural low-dimensional version of state augmentation.

Baselines

The naive reference is DPVO unchanged, with per-patch diagonal weights, trained and evaluated with its released configuration. No reported absolute trajectory error for DPVO on EuRoC or TartanAir was found on Lacuna; the file reports only that it has 40% lower error than DROID-SLAM. We will therefore rerun it ourselves.

The strongest existing method is MAC-VO. The file reports a 61.9% improvement in relative translation error over DROID-SLAM on TartanAir v2 Hard and an 82.4% reduction on the texture-less Lunar split, with no absolute values. These gains are relative to DROID-SLAM, not relative to a diagonal-covariance version of MAC-VO, so they do not by themselves show what off-diagonal structure buys. We will run MAC-VO from its release on the same splits.

Our head is added to DPVO first. We will also add the inter-patch low-rank term to MAC-VO's keypoints, to test whether the gain holds on top of the strongest system. Beating MAC-VO means lower relative translation error on TartanAir v2 Hard and Lunar, at matched training data, without worse normalized estimation error.

Three ablations isolate the key ingredient. First, global module, diagonal output: the same criss-cross block feeds only $D$. This separates "better context for per-patch variance" from "correlation structure". Second, local-only: $D + K_{\ell}$ with no low-rank term. Third, low-rank from local features: $L$ is predicted by a convolutional head with the same parameter count but no non-local routing. This last ablation is the direct test of the receptive-field claim.

Two further competitors complete the comparison. One is a covariance-intersection-style fusion of patch groups. The other is scalar covariance inflation fitted on a validation split. All variants share the backbone, the TartanAir training data, the iteration count and the hyperparameter search budget, with 20 trials each on a held-out TartanAir split.

Why not just…?

Why not just inflate the covariance by a fitted scalar? Multiplying all patch covariances by a constant leaves the weighted least-squares pose solution unchanged. It can therefore repair overconfidence but cannot remove bias from a shared error mode. Our experiments report accuracy and calibration separately, so scalar inflation should match us on calibration in easy sequences and fail to match us on trajectory error. If it matches on both, the correlation structure is not doing work, and we will say so.

Why not just sample patches farther apart, as DPVO already does? This is the most serious threat to the local term. Spatial thinning plausibly removes most receptive-field-overlap correlation for free. It cannot remove correlation from blur or exposure changes that affect the whole frame. We will run DPVO with enforced minimum patch spacing as a baseline. If it matches our local-only variant, we will drop $K_{\ell}$ and keep only the global modes, which is where we expect the result to stand or fall.

Why not just give the uncertainty head a bigger receptive field, for example with dilated convolutions or a deeper backbone, and keep a diagonal output? A larger receptive field lets each patch's variance reflect global conditions. But a diagonal output still says nothing about whether two patches err in the same direction, which is what the solver needs to avoid double counting. The "global module, diagonal output" ablation tests exactly this. If it captures most of the gain, the receptive-field claim survives but the covariance claim does not.

Why not just bound the unknown correlations with covariance intersection? Covariance intersection, as used in DInCIKF, guarantees consistency under unknown correlation at the cost of conservative weights and slower convergence. We expect it to be well calibrated but less accurate. Including it shows whether modeling correlation beats bounding it.

Proposed Experiments and Decisive Checks

Training uses TartanAir. Evaluation covers TartanAir v2, including the Hard and Lunar splits, EuRoC MAV for real-world zero-shot transfer, and KITTI odometry for long driving sequences. Metrics are absolute trajectory error, relative translation and rotation error, and normalized estimation error squared (NEES), the squared pose error scaled by the predicted covariance, whose expected value equals the state dimension when uncertainty is honest. We will also report runtime and memory overhead against DPVO's 60 FPS default.

To make the mechanism visible, we add a controlled stress test. TartanAir sequences are perturbed with synthetic global disturbances: frame-wide motion blur, exposure flicker, and a moving occluder covering up to 30% of the view. Each perturbation is applied at increasing strength, so we can check whether the learned modes in $L$ align with the injected cause and whether our gains grow with disturbance strength.

The first decisive check is the measurement study. If inter-patch error correlation at large distances is near zero on TartanAir v2, the long-range claim is false and the project reduces to a calibration note. The second is the "low-rank from local features" ablation. If a convolutional head predicts the shared modes as well as the non-local one, the link between receptive field and covariance does not hold. The third is calibration: our method should bring NEES close to its nominal value on EuRoC, where DPVO-style diagonal weights should be overconfident. A trajectory-error gain without a calibration gain, or the reverse, would tell us which half of the story is real.