ν-DBA: Neural Implicit Dense Bundle Adjustment Enables Image-Only Driving Scene Reconstruction

Yunxuan Mao, Bingqi Shen, Yifei Yang, Kai Wang, Rong Xiong, Yiyi Liao, Yue Wang

Introduction

Reconstructing dense, accurate 3D maps of driving environments using only camera data remains a primary challenge in autonomous vehicle research. While traditional methods like Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM) have made significant strides, they often struggle with global consistency and the inherent noise of outdoor environments. ν-DBA (Neural Implicit Dense Bundle Adjustment) introduces a novel framework that bridges the gap between 3D-consistent neural implicit surfaces and the geometric robustness of dense optical flow, offering a superior alternative to traditional photometric-based reconstruction.

The Challenges of Outdoor Reconstruction

Two major hurdles have historically prevented high-fidelity, image-only reconstruction in driving scenarios:

  1. Map Consistency: Traditional dense Bundle Adjustment (BA) methods, such as DROID-SLAM, typically optimize depth maps for individual keyframes. This approach lacks a global 3D constraint, leading to discrepancies where the same spatial point may be assigned different coordinates across different frames.
  2. The Photometric Trap: Many recent neural reconstruction methods, including Neural Radiance Fields (NeRF), rely on minimizing photometric error—the difference between rendered and actual pixel colors. In outdoor driving, this metric often fails due to complex lighting shifts, reflections, and large textureless surfaces like asphalt roads, which provide ambiguous signals for optimization.

ν-DBA addresses these issues by replacing color-based optimization with a geometric error metric derived from dense optical flow and utilizing a neural implicit surface for unified 3D consistency.

The ν-DBA Framework

The core innovation of ν-DBA lies in its integration of several key components into a cohesive Bundle Adjustment pipeline:

Neural Implicit Map Parametrization

Instead of relying on discrete point clouds or 2D depth maps, ν-DBA utilizes a Close-Range SDF (Signed Distance Function) Field. This field is implemented via a 3D hash table and a shallow Multi-Layer Perceptron (MLP). This architecture ensures a single, continuous, and 3D-consistent representation of the environment, naturally regularizing the geometry across multiple viewpoints.

Geometric Dense Bundle Adjustment

Rather than comparing pixel colors, the system minimizes the distance between the projected 3D surface and the dense optical flow predicted between frames. This geometric approach is significantly more stable in outdoor environments, as optical flow is less sensitive to the lighting fluctuations and reflections that typically plague photometric methods.

Self-Supervised Flow and Initialization

To maximize accuracy, the framework fine-tunes the underlying optical flow model (Unimatch) using per-scene self-supervision. This narrows the generalization gap inherent in pre-trained models. Furthermore, to handle the "unbounded" nature of outdoor scenes, the authors employ RANSAC on triangulated flow points to identify the ground plane early, providing a vital geometric prior for the neural model.

Decoupled Rendering

In ν-DBA, geometry is the priority. The system optimizes the 3D structure first using geometric cues. Color rendering is treated as an optional, decoupled module that is only trained if novel view synthesis is required, ensuring that noisy color data does not interfere with the geometric reconstruction.

Experimental Evaluation

The framework was rigorously tested on three distinct datasets: KITTI-360, the Waymo Open Dataset, and a specialized Mine Truck Dataset. The latter features unstructured views and small obstacles (e.g., 30cm high convex rocks) that are notoriously difficult for standard SLAM systems to detect.

Key Results

  • Trajectory Accuracy: On the KITTI-360 benchmark, ν-DBA achieved an Absolute Trajectory Error (ATE) of 0.073m, outperforming established methods like DROID-SLAM (0.084m) and StreetSurf (0.078m).
  • Reconstruction Fidelity: The system nearly doubled the mesh accuracy of DROID-SLAM on KITTI-360, reaching 41.63cm compared to the baseline's 87.01cm.
  • Robustness to Small Obstacles: In the Mine Truck dataset, ν-DBA successfully reconstructed small obstacles that were either blurred or entirely missed by competing neural implicit methods.
  • Efficiency: Despite its complexity, the model can be trained in approximately 30 minutes on a single NVIDIA RTX 4090, making it highly competitive for high-accuracy back-end reconstruction tasks.

Discussion: Geometry Over Photometry

One of the most significant insights from this research is that the inclusion of photometric loss actually degraded geometric accuracy. The authors demonstrate that color-based losses introduce noise and ambiguity that conflict with the clean geometric signals provided by optical flow. By treating flow as a universal proxy for pixel movement, ν-DBA avoids the scale ambiguity issues common in monocular depth estimation and provides a direct, robust link between 2D movement and 3D structure.

Limitations and Future Work

While ν-DBA sets a new standard for accuracy, it is currently designed as a high-fidelity back-end rather than a real-time SLAM system. Its performance is also dependent on the quality of the optical flow; in extremely low-light or featureless environments where flow estimators struggle, reconstruction quality may degrade. Future research will focus on extending the framework to handle dynamic objects, such as moving vehicles and pedestrians, within the dense BA pipeline.

Conclusion

ν-DBA represents a significant step forward in autonomous scene reconstruction. By prioritizing geometric consistency through neural implicit surfaces and dense optical flow, it overcomes the "photometric trap" and provides a robust, accurate solution for mapping complex outdoor environments from images alone.

Summary updated

Abstract

The joint optimization of the sensor trajectory and 3D map is a crucial characteristic of bundle adjustment (BA), essential for autonomous driving. This paper presents ν-DBA, a novel framework implementing geometric dense bundle adjustment (DBA) using 3D neural implicit surfaces for map parametrization, which optimizes both the map surface and trajectory poses using geometric error guided by dense optical flow prediction. Additionally, we fine-tune the optical flow model with per-scene self-supervision to further improve the quality of the dense mapping. Our experimental results on multiple driving scene datasets demonstrate that our method achieves superior trajectory optimization and dense reconstruction accuracy. We also investigate the influences of photometric error and different neural geometric priors on the performance of surface reconstruction and novel view synthesis. Our method stands as a significant step towards leveraging neural implicit representations in dense bundle adjustment for more accurate trajectories and detailed environmental mapping.
BibTeX
@inproceedings{mao2024dba,
  author = {Yunxuan Mao and Bingqi Shen and Yifei Yang and Kai Wang and Rong Xiong and Yiyi Liao and Yue Wang},
  title = {{ν-DBA: Neural Implicit Dense Bundle Adjustment Enables Image-Only Driving Scene Reconstruction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=S9r7N29R9O},
}
Figure