Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction

Tristan Laidlow, Jan Czarnowski, Andrea Nicastro, Ronald Clark, Stefan Leutenegger

TL;DR

This paper proposes a novel method to fuse deep learning-based depth predictions with traditional 3D reconstruction pipelines. By treating depth as a probability distribution and combining it with photometric consistency cues, the system achieves more robust and accurate 3D reconstructions, particularly in challenging areas like textureless regions or occlusions.

What Problem Does This Paper Address?

Creating accurate and dense 3D reconstructions from images is a long-standing challenge in computer vision. Traditional methods like Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM) often struggle with ambiguities arising from textureless surfaces, occlusions, and repetitive patterns. While deep learning has shown great promise in predicting depth from single images, integrating these learned "priors" into existing geometric pipelines in a principled and probabilistic way remains an open problem. Simply feeding raw network outputs into traditional systems can lead to brittle results, and understanding the uncertainty associated with these predictions is crucial for applications like robotics.

Methods

The core of this work lies in a probabilistic fusion approach. Instead of predicting a single depth value, their deep neural network (DNN) outputs a discrete, non-parametric probability distribution for the depth of each pixel. This "probability volume" captures uncertainty and allows for multi-hypothesis predictions.

Here's a breakdown of their method:

  • Learned Depth Prior: A ResNet-50 based DNN is trained to predict these per-pixel depth probability distributions from a single image. They use an "ordinal loss" function to penalize predictions that are close but incorrect less than those that are far off.
  • Photometric Consistency: For each new frame, they compute a traditional photometric cost volume based on pixel intensity differences across multiple views. This is then converted into a probability volume.
  • Probabilistic Fusion: The learned depth prior probability volume and the photometric probability volume are fused element-wise using multiplication, followed by normalization. This fusion is key to leveraging the strengths of both approaches.
  • Kernel Density Estimation (KDE): To smooth out the discrete probability distributions and enable gradient-based optimization, they apply KDE.
  • Regularization: The final depth map is extracted by minimizing a cost function that includes a unary term (based on the smoothed PDF) and a pairwise term. Crucially, this pairwise term uses surface normals and occlusion boundaries predicted by another DNN (SharpNet) to preserve fine geometric details and avoid smoothing across discontinuities.
  • Keyframe Warping: To maintain consistency across frames and avoid re-initializing from scratch, they warp the probability volume of the previous keyframe into the new frame. This involves converting the depth probability to an occupancy probability, warping it, and converting it back.

Datasets

  • Training: The paper mentions training their depth prediction network on the ScanNet RGB-D dataset.
  • Evaluation: The system is evaluated on sequences from the TUM RGB-D dataset. The ground truth depth from this dataset is used solely for validation.

Key Insights and Ideas

  • Probabilistic Fusion is Key: The paper strongly advocates for treating learned predictions as probability distributions rather than single values. This allows for a principled fusion with geometric cues, leading to better results than either source alone.
  • Complementary Strengths: The fusion effectively combines the DNN's ability to handle textureless regions and occlusions with the photometric method's precision on high-gradient areas.
  • DNN-predicted Regularization: Using network-predicted surface normals and occlusion boundaries for regularization is a powerful way to preserve fine geometric details that traditional smoothness priors might blur.
  • Warping Probability Volumes: The novel approach to warping probability volumes between keyframes is crucial for maintaining temporal consistency and improving performance over time.

Key Results

  • Improved Accuracy: In six out of eight tested sequences, the fused system outperforms using either the network-only or photometric-only probability volumes.
  • Robustness: The fused system shows significant improvements in handling textureless regions and occlusions, as demonstrated qualitatively in Figure 6 and quantitatively in Table I.
  • Regularization Benefits: Table II shows that using normals and occlusions for regularization consistently yields the best results compared to simpler regularization schemes or no optimization.
  • Warping Benefits: Table III demonstrates that warping probability volumes between keyframes consistently improves reconstruction accuracy.
  • Qualitative Improvements: Figure 5 and 6 visually showcase how the fused probability distributions resolve ambiguities and lead to more coherent depth maps.

Limitations and Future Work

  • Not Real-Time: The current implementation is not real-time, with the optimization step taking up to 1.2 seconds per frame. The authors suggest that using methods like Newton's method or primal-dual algorithms could significantly speed this up.
  • Specific Dataset Performance: The paper notes that for the fr1/floor sequence, the network's priors were particularly poor because the scene was outside its training distribution, leading to the photometric-only approach performing best. This highlights the dependence of learned priors on the training data.
  • Future Work: The authors suggest exploring faster optimization methods and potentially more sophisticated ways to fuse the probability volumes. They also acknowledge that while their system doesn't achieve the absolute highest accuracy of some end-to-end deep learning systems (like DeepTAM), it maintains a crucial probabilistic formulation for applications like robotics.

Summary updated

Abstract

The best way to combine the results of deep learning with standard 3D reconstruction pipelines remains an open problem. While systems that pass the output of traditional multi-view stereo approaches to a network for regularisation or refinement currently seem to get the best results, it may be preferable to treat deep neural networks as separate components whose results can be probabilistically fused into geometry- based systems. Unfortunately, the error models required to do this type of fusion are not well understood, with many different approaches being put forward. Recently, a few systems have achieved good results by having their networks predict probability distributions rather than single values. We propose using this approach to fuse a learned single-view depth prior into a standard 3D reconstruction system. Our system is capable of incrementally producing dense depth maps for a set of keyframes. We train a deep neural network to predict discrete, nonparametric probability distributions for the depth of each pixel from a single image. We then fuse this "probability volume" with another probability volume based on the photometric consistency between subsequent frames and the keyframe image. We argue that combining the probability volumes from these two sources will result in a volume that is better conditioned. To extract depth maps from the volume, we minimise a cost function that includes a regularisation term based on network predicted surface normals and occlusion boundaries. Through a series of experiments, we demonstrate that each of these components improves the overall performance of the system.
BibTeX
@inproceedings{laidlow2020towards,
  author = {Tristan Laidlow and Jan Czarnowski and Andrea Nicastro and Ronald Clark and Stefan Leutenegger},
  title = {{Towards the Probabilistic Fusion of Learned Priors into Standard Pipelines for 3D Reconstruction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=i0tbV_qpdc},
}
Figure