BA-Net: Dense Bundle Adjustment Networks

Chengzhou Tang, Ping TanICLR 2019 · OpenReview

TL;DR

BA-Net bridges the gap between classical multi-view geometry and deep learning by embedding Bundle Adjustment (BA) as a fully differentiable layer within a neural network. By optimizing a novel "feature-metric" error and using a compact basis-depth parameterization, the system achieves state-of-the-art performance in dense Structure-from-Motion (SfM) while remaining end-to-end trainable.

What Problem Does This Paper Address?

Structure-from-Motion (SfM)—the process of estimating 3D structures and camera motion from image sequences—has traditionally relied on Bundle Adjustment (BA). Standard BA comes in two flavors:

  1. Geometric BA: Minimizes the re-projection error of sparse points. It is highly accurate but relies on error-prone feature matching and ignores the vast majority of image data.
  2. Photometric BA: Minimizes pixel intensity differences. It utilizes more data but is notoriously sensitive to lighting changes, moving objects, and poor initializations.

Recent deep learning approaches often treat SfM as a black-box regression problem, frequently failing to enforce the hard geometric constraints that make classical methods so robust. BA-Net aims to combine the best of both worlds: the rigorous geometric constraints of classical BA and the powerful representation learning capabilities of deep networks.

Methods

The authors introduce three major innovations to make this hybrid approach work:

BA-Net Architecture
  • The BA-Layer: This is a differentiable implementation of the Levenberg-Marquardt (LM) algorithm. To make the optimization differentiable, the authors use a Multi-Layer Perceptron (MLP) to predict the "damping factor" ($\lambda$) at each iteration. This allows the network to backpropagate gradients through the entire optimization process.
  • Feature-metric BA: Instead of aligning raw pixels (photometric) or sparse points (geometric), BA-Net aligns dense CNN feature maps. Because these features are learned specifically to make the optimization landscape smoother, the "basin of attraction" is much wider, leading to better convergence and robustness against lighting or motion noise.
  • Basis Depth Maps: Optimizing a depth value for every single pixel is computationally impossible within a network layer. BA-Net solves this by generating a small set (e.g., 128) of "basis depth maps" for an image. The final dense depth is a linear combination of these bases. This reduces the number of variables from hundreds of thousands to just a few weights, making dense BA tractable.

Datasets

The model was evaluated on several diverse benchmarks:

  • ScanNet: A large-scale indoor dataset with over 1,500 sequences.
  • KITTI: A standard outdoor driving dataset used to evaluate Absolute Trajectory Error (ATE) and depth.
  • DeMoN Dataset: A collection including MVS, Scenes11 (synthetic), RGB-D, and Sun3D datasets.
  • EuroC: Used specifically for comparison against other depth parameterization methods like CodeSLAM.

Key Insights and Ideas

  • Optimization as a Layer: By treating the optimizer itself as a neural network layer, the model doesn't just learn features; it learns features that are easy for an optimizer to use.
  • Inductive Bias Matters: Hard-coding multi-view geometry constraints into the network architecture (via the BA-Layer) acts as a powerful inductive bias, allowing the model to generalize better than pure regression-based "black box" models.
  • Smoothing the Landscape: Visualization of the feature-metric error shows a much smoother objective function with a clear global minimum compared to raw RGB values, which are plagued by local minima.
Objective Function Landscape Comparison
  • Compact Parameterization: The basis depth map approach ensures that the resulting depth maps are naturally smooth and consistent with object boundaries, as the bases are generated by a CNN that understands image structure.

Key Results

  • Superior Accuracy: On ScanNet, BA-Net achieved a rotation error of 1.018°, significantly outperforming the popular DeMoN network (3.791°) and classical BA baselines.
  • Outdoor Performance: On KITTI, BA-Net outperformed both supervised (Eigen et al.) and unsupervised (Zhou et al., Godard et al.) methods in both trajectory accuracy and depth metrics.
  • Efficiency: Despite the complex optimization happening inside the network, BA-Net runs in approximately 95.21 ms for a pair of images, making it faster than DeMoN (~110 ms).
  • Multi-view Scaling: The accuracy consistently improves as more views (up to 5) are added, proving that the BA-Layer effectively utilizes multi-view constraints.

Limitations and Future Work

  • Computational Bottleneck: The current bottleneck is the BA-Layer itself, which involves heavy matrix operations. The authors suggest that a dedicated CUDA implementation could further speed this up.
  • Basis Map Dependency: The quality of the final depth is capped by the quality of the generated basis maps. If the generator fails to capture a specific geometric detail, the BA-Layer cannot recover it.
  • Future Directions: The authors propose that the backbone (DRN-54) could be shared with other tasks like semantic segmentation or object detection to create a more holistic scene-understanding system.

Summary updated

Abstract

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.
BibTeX
@inproceedings{tang2019ba,
  author = {Chengzhou Tang and Ping Tan},
  title = {{BA-Net: Dense Bundle Adjustment Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=U84wec2AiuI},
}
Figure