Dense 3D-Reconstruction From Monocular Image Sequences for Computationally Constrained UAS

Matthias Domnik, Pedro Proenca, Jeff Delaune, Jorg Thiem, Roland BrockersWACV 2021

TL;DR

This paper presents a novel Structure-from-Motion (SfM) approach for generating dense 3D reconstructions from monocular downward-facing cameras on computationally limited Unmanned Aerial Systems (UAS). By leveraging metric pose priors from a Visual-Inertial Odometry (VIO) system and employing a windowed Bundle Adjustment (BA) for pose refinement, the method achieves real-time performance suitable for autonomous landing site detection on platforms like the Mars Helicopter.

What Problem Does This Paper Address?

Autonomous navigation for Unmanned Aerial Systems (UAS), especially in unknown terrain, requires robust on-board 3D perception for tasks like safe landing site identification. This is particularly challenging for small UAS that have strict limitations on size, weight, and computational power. While traditional SfM methods can produce accurate 3D reconstructions, they are often computationally intensive and not suitable for real-time execution on embedded systems. Existing real-time methods often produce sparse reconstructions, which are insufficient for detailed terrain analysis. The paper aims to bridge this gap by enabling dense 3D reconstruction on resource-constrained platforms.

Methods

The proposed approach combines several key components:

  • Monocular Downward-Facing Camera: The system relies on a single camera looking downwards, simplifying the setup and reducing computational load compared to stereo or multi-camera systems.
  • Visual-Inertial Odometry (VIO) for Pose Priors: Instead of relying solely on visual information for pose estimation (which can be scale-ambiguous and drift over time), the system uses metric pose estimates from an existing VIO estimator (specifically, the xVIO estimator). This provides a crucial initial guess for camera poses, including scale information.
  • Feature Tracking: A KLT-based tracker is used to extract and track FAST features across image frames. To ensure an even distribution of features, image binning is employed. Lost tracks are replenished with newly detected features.
  • Keyframe Management: A sliding window buffer is used to manage keyframes. Keyframes are selected based on image overlap and a minimum parallax constraint to ensure sufficient baseline for reconstruction. The selection aims to preserve existing information while introducing new observations.
  • Windowed Bundle Adjustment (BA) for Pose Refinement: A core innovation is the use of a windowed BA. Instead of optimizing all past camera poses, it refines only the poses within a recent history (a "window"). This significantly reduces computational cost while still improving accuracy for the poses relevant to current reconstruction. The objective function minimizes reprojection error, incorporating both feature observations and VIO pose priors. A Huber norm is used for robustness against outliers.
  • Dense Stereo Reconstruction: After pose refinement, a pair of keyframes with an optimal baseline and sufficient overlap is selected. These frames are then rectified, and a standard block-matching stereo algorithm is applied to generate a dense 3D point cloud.

Datasets

The paper does not introduce new datasets. Instead, it utilizes:

  • Simulated Data: The authors employ a photo-realistic 3D simulator ([15]) to generate synthetic flight data. This allows for controlled experiments with ground truth camera poses and the ability to introduce various noise models (feature observation noise, camera pose perturbations, IMU noise).
  • Real-World UAS Flight Data: The approach is validated using data collected from UAS flights over a desert area.

Key Insights and Ideas

  • Leveraging VIO for Scale and Initial Guess: The paper highlights the critical role of VIO in providing metric scale and accurate initial pose estimates. This significantly simplifies the SfM problem and enables the use of less computationally intensive BA.
  • Windowed BA for Real-Time Performance: The key insight for achieving real-time performance on constrained hardware is the windowed BA. By limiting the optimization to a small subset of recent keyframes, the computational burden is drastically reduced without sacrificing too much accuracy for immediate reconstruction needs.
  • Robustness through Huber Loss: The use of the Huber norm in the BA objective function is crucial for handling outliers in feature observations, which are common in real-world scenarios and can destabilize optimization.
  • Metric VEE as a Key Evaluation Metric: The paper introduces and emphasizes the Vertical Epipolar Error (VEE) as a critical metric for evaluating the accuracy of camera pose refinement, as it directly impacts the success of subsequent stereo matching.

Key Results

  • Accuracy: The proposed method, with pose refinement, achieves a median VEE below 0.25 px in over 74% of scenarios in simulations, which is deemed sufficient for dense stereo reconstruction. Real-world flight data also demonstrates successful dense 3D reconstruction.
  • Speed: The windowed BA approach, particularly when using the Ceres optimization framework, is shown to be significantly faster than full BA or other frameworks like g2o and GTSAM, making it suitable for real-time applications on embedded systems. The run time per iteration grows exponentially with window length, leading to the selection of a 7-frame window as a good trade-off.
  • Outlier Rejection: An outlier rejection scheme based on reprojection error significantly improves the median VEE, reducing instances of VEEs above 0.25 px from over 71% to less than 5% in simulated flights.

Limitations and Future Work

  • Limitations:
    • The performance is still influenced by the quality of the VIO pose priors.
  • The system relies on sufficient texture in the scene for feature tracking.
  • Occluded regions and moving shadows can still lead to gaps in the reconstruction.
  • Future Work:
    • Improved Outlier Rejection: Further development of outlier rejection mechanisms is suggested to enhance accuracy.
  • Covariance-Based Rejection: Estimating the lower bound of achievable accuracy using the covariance matrix of the approximated Hessian could provide an additional criterion for rejecting ill-posed problems.
  • Integration with Probabilistic Mapping: The generated dense depth maps are intended to be absorbed by subsequent probabilistic mapping approaches for tasks like landing site selection.

Summary updated

BibTeX
@inproceedings{domnik2021dense,
  author = {Matthias Domnik and Pedro Proenca and Jeff Delaune and Jorg Thiem and Roland Brockers},
  title = {{Dense 3D-Reconstruction From Monocular Image Sequences for Computationally Constrained UAS}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure