From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers

Swaminathan Gurumurthy, Karnik Ram, Bingqing Chen, Zachary Manchester, Zico KolterCVPR 2024 · DOI · arXiv

Abstract

Training state-of-the-art visual odometry and SLAM models that utilize differentiable Bundle Adjustment (BA) layers is notoriously slow and unstable. This paper identifies high gradient variance as the primary culprit and proposes a simple "weighted flow loss" that leverages the network's own internal confidence estimates to filter out noisy outliers during training. This approach results in a 2–2.5× training speedup and significantly improved accuracy across standard benchmarks.

Introduction

In recent years, top-performing robotics models for pose estimation, such as DROID-SLAM and DPVO, have adopted a "learning to optimize" approach. These systems combine deep neural networks for feature extraction and correspondence with differentiable Bundle Adjustment (BA) layers that enforce geometric constraints. While highly accurate, these models are difficult to train; for instance, DROID-SLAM can take over a week to train on multiple high-end GPUs. This sluggishness stems from high gradient variance, which forces researchers to use tiny learning rates and various training heuristics to prevent instability.

Unbundling Gradient Variance

The authors analyze the mathematical foundations of the BA layer and identify three specific causes of gradient noise that hinder training:

  1. Flow Loss Interference: Gradients from noisy outlier points can overwhelm the signal from reliable inlier points.
  2. Linearization Errors: BA layers rely on Gauss-Newton steps, which approximate complex geometry as linear. If initial depth or pose estimates are poor, this approximation fails, injecting noise into the backpropagation process.
  3. Weight-Residue Dependency: The gradients for confidence weights are directly tied to optimization residuals, causing outliers to dominate weight updates.
Sources of gradient variance in differentiable Bundle Adjustment layers.

The Weighted Flow Loss Solution

The proposed solution is a "weighted flow loss" mechanism. The network already predicts confidence weights ($\Sigma$) for every point to use within the BA optimization. The authors suggest using these same weights to scale the outer training loss (the flow loss).

By applying a "stop gradient" to these weights—ensuring the loss doesn't simply push weights to zero to minimize itself—the training objective is forced to focus on points the model is actually confident about. Additionally, a dynamic balancing coefficient ($\beta$) is introduced to automatically adjust the ratio between pose loss and flow loss throughout the training cycle.

Key Insights and Results

A central insight of this work is that the network's internal confidence measures are ideal "denoisers" for the training signal itself. The paper quantifies how sensitive BA gradients are to noise in initial estimates, showing that the Signal-to-Noise Ratio (SNR) in gradients drops significantly as noise increases.

Gradient error norm increases significantly with added pose noise, illustrating linearization sensitivity.

The researchers evaluated their approach on benchmarks including TartanAir, EuRoC, and TUM-RGBD. Key results include:

  • Massive Speedup: A modified DPVO model reached peak accuracy in 80k iterations, compared to the original 180k—a 2.25× speedup.
  • Improved Accuracy: On the TartanAir test set, the proposed method achieved a 24% lower Average Trajectory Error (ATE) than the baseline.
  • Robustness: The model became less sensitive to hyperparameters and demonstrated improved convergence speed when applied to DROID-SLAM.

Conclusion

By mitigating gradient variance, the weighted flow loss significantly stabilizes and accelerates the training of differentiable BA layers. While challenges like the sim-to-real gap and potential trade-offs in depth accuracy for outliers remain, this work provides a robust framework for improving the practicality of "learning to optimize" models in robotics.

Summary updated

BibTeX
@inproceedings{gurumurthy2024from,
  author = {Swaminathan Gurumurthy and Karnik Ram and Bingqing Chen and Zachary Manchester and Zico Kolter},
  title = {{From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02597},
}
Figure