Burst Imaging for Light-Constrained Structure-From-Motion

Ahalya Ravendran, Mitch Bryson, Donald G. Dansereau

TL;DR

This paper tackles the challenge of 3D reconstruction in extremely low-light conditions, where traditional computer vision methods fail due to noise. They propose a novel approach that leverages burst photography to create higher-quality images, significantly improving the accuracy and robustness of Structure-from-Motion (SfM) pipelines for robots operating in challenging environments.

What Problem Does This Paper Address?

Robots increasingly rely on vision for tasks like navigation, mapping, and object recognition. However, many of these applications, such as autonomous driving at night or operating in underground mines, involve extremely low-light conditions. In such scenarios, images suffer from low Signal-to-Noise Ratio (SNR), leading to noisy and unreliable features. This noise causes existing state-of-the-art Structure-from-Motion (SfM) algorithms to fail, resulting in inaccurate 3D reconstructions, incorrect camera trajectories, and even complete convergence failures. The core problem is bridging the gap between the need for robust visual perception in low light and the limitations of current SfM techniques when faced with noisy input.

Methods

The authors propose a pipeline that combines burst photography with feature-based SfM. The key innovation lies in how they process burst images for reconstruction:

  • Burst Acquisition: Instead of single images, they capture a rapid sequence of short-exposure images (a "burst") of the same scene.
  • Hierarchical Tile-Based Alignment: To handle motion within a burst, they employ a coarse-to-fine alignment strategy using Gaussian pyramids. This aligns individual "tiles" of consecutive images to a common reference tile, minimizing pixel intensity errors.
  • Robust Temporal Merge: After alignment, the burst images are merged into a single, denoised image. They use a voting scheme and temporal filtering (with options for Wiener or bilateral filtering) to create a robust merged image, even if some individual frames have alignment issues.
  • Feature-Based SfM: The resulting denoised images from multiple bursts are then fed into a standard SfM pipeline (COLMAP). The improved image quality leads to more accurate feature detection, matching, and ultimately, a more robust and accurate 3D reconstruction and camera trajectory.

This approach differs from conventional methods by actively processing multiple frames to combat noise, and from typical burst photography applications by focusing on reconstruction quality rather than just visual aesthetics.

Datasets

The authors created their own dataset for evaluation. They mounted a monocular machine vision camera on a UR5e robotic arm and captured bursts of images in an illumination-controlled environment. This dataset consists of:

  • 20 scenes with objects of varying textures, shapes, and sizes.
  • 22 bursts per scene, yielding 6160 images in total.
  • Images captured at two different exposure times: 1ms and 0.1ms, with gain adjusted to maximize contrast while avoiding saturation.
  • Raw Bayer images were captured.

They are releasing this dataset and associated code to the research community.

Key Insights and Ideas

  • Burst Merging is Crucial for SfM: Simply averaging burst images without proper alignment (naïve averaging) doesn't solve the problem and can even introduce motion blur. The proposed alignment and merging process is essential for extracting meaningful information for SfM.
  • Higher SNR = More Robust Features: The core benefit of burst merging is the significant improvement in SNR. This directly translates to detecting more "true" features and fewer "spurious" features, which is critical for SfM algorithms.
  • Reduced Computational Burden: While merging adds an initial processing step, the resulting higher-quality images lead to fewer outliers and less need for extensive feature rejection in the SfM pipeline, potentially reducing overall computation.
  • Low Light is a Major Bottleneck for Robotics: The paper highlights that current SfM methods are highly sensitive to lighting, severely limiting robotic applications in many real-world scenarios. This work offers a path to overcome this limitation.

Key Results

  • Improved Feature Performance: In synthetic tests with varying noise levels, their "burst with merge" method consistently outperformed conventional noisy images and "burst without merge" in terms of true positive (TP) rate and false positive (FP) count. Even in strong noise, their method detected all true features, while others failed.
  • Enhanced Reconstruction Accuracy: Quantitative results (Table I) show that their method, when using COLMAP with default settings, reconstructs significantly more registered images per input scene, generates more 3D points per image, and achieves higher match ratios and precision compared to conventional methods.
  • Robustness to Low Light: At 0.1ms exposure (higher noise), their method successfully reconstructed all scenes, even when conventional methods failed or required parameter tuning.
  • Accurate Camera Trajectories: Figure 8 and Table II demonstrate that their method produces more accurate camera trajectories with lower translation and rotation errors compared to conventional approaches, especially in challenging low-light conditions.
  • Efficiency: Their method uses fewer images (22 bursts of 7 images) and takes significantly less time (3.25 minutes) for reconstruction compared to "burst without merge" (154 images, 49.58 minutes) on average.

Limitations and Future Work

  • Limitations:
    • The method assumes that the underlying signal is not completely suppressed by quantization noise.
  • Sufficient overlap between subsequent bursts is required for registration.
  • While the paper discusses trade-offs, the optimal parameters for burst capture (number of frames, exposure time, gain) are application-dependent.
  • Future Work:
    • The authors suggest exploring adaptive sampling schemes where burst capture and processing parameters are dynamically adjusted to suit the situation.
  • They also propose fusing complementary sensors to further enhance performance.

Summary updated

Abstract

Images captured under extremely low light conditions are noise-limited, which can cause existing robotic vision algorithms to fail. In this letter we develop an image processing technique for aiding 3D reconstruction from images acquired in low light conditions. Our technique, based on burst photography, uses direct methods for image registration within bursts of short exposure time images to improve the robustness and accuracy of feature-based structure-from-motion (SfM). We demonstrate improved SfM performance in challenging light-constrained scenes, including quantitative evaluations that show improved feature performance and camera pose estimates. Additionally, we show that our method converges more frequently to correct reconstructions than the state-of-the-art. Our method is a significant step towards allowing robots to operate in low light conditions, with potential applications to robots operating in environments such as underground mines and night time operation.
BibTeX
@article{ravendran2022burst,
  author = {Ahalya Ravendran and Mitch Bryson and Donald G. Dansereau},
  title = {{Burst Imaging for Light-Constrained Structure-From-Motion}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=E9yYcG-oSF},
}
Figure