Smooth Mesh Estimation from Depth Data using Non-Smooth Convex Optimization

Antoni Rosinol, Luca Carlone

TL;DR

This paper presents a novel method for reconstructing accurate and smooth 3D meshes directly from depth maps and sparse visual odometry landmarks. By formulating the problem as a non-smooth convex optimization task and solving it with a primal-dual algorithm, they achieve real-time performance while significantly improving mesh quality over existing methods.

What Problem Does This Paper Address?

Creating accurate and efficient 3D maps of environments is crucial for robotics and augmented reality applications. While 3D meshes are ideal due to their lightweight nature and ability to encode scene topology, directly generating them from sensor data is challenging. Traditional methods often rely on computationally expensive volumetric representations or struggle with accuracy when directly meshing sparse visual odometry features. This paper aims to bridge this gap by directly constructing a high-quality 3D mesh from readily available depth data and sparse visual landmarks, bypassing intermediate representations and achieving real-time performance.

Methods

The core of this work lies in formulating the 3D mesh reconstruction as a non-smooth convex optimization problem. The objective function balances three key aspects:

  1. Data Fitting (Ldepth): Ensuring the reconstructed mesh accurately represents the input depth map.
  2. Landmark Tracking (Ltracking): Aligning the mesh vertices with sparse 3D landmarks provided by a visual odometry (VO) or visual-inertial odometry (VIO) pipeline.
  3. Mesh Smoothing (Lsmooth): Enforcing piecewise planarity and smoothness to handle noise and outliers in the depth data. The authors leverage the second-order Total Generalized Variation (TGV2) functional, adapted for discrete meshes, to achieve this.

To handle the non-smoothness introduced by the L1 norms in the data fitting and smoothing terms, they employ a first-order primal-dual optimization method. This approach iteratively updates primal variables (mesh vertex positions and normals) and dual variables (related to the optimization constraints) until convergence. A key implementation detail is the use of a GPU for accelerating the computation of the dual updates, which is crucial for achieving real-time performance. They also introduce "Steiner points" – additional mesh vertices not directly derived from VO landmarks – to improve the mesh's ability to fit complex geometry.

Datasets

The paper evaluates its approach on two datasets:

  • EuRoC dataset: This dataset provides visual and inertial data from a micro aerial vehicle. For this work, dense stereo reconstruction was performed on the RGB-D data to obtain depth maps for mesh fitting.
  • uHumans2 dataset: This dataset consists of synthetic RGB-D data and ground truth geometry, generated using a photorealistic simulator. It was used to evaluate the accuracy of mesh reconstruction and to perform ablation studies on the impact of Steiner points.

Key Insights and Ideas

  • Direct Mesh Reconstruction is Feasible and Efficient: The paper demonstrates that it's possible to bypass computationally intensive volumetric methods and directly reconstruct high-quality 3D meshes from depth and sparse landmarks in real-time.
  • Convex Optimization as a Powerful Tool: Formulating mesh reconstruction as a non-smooth convex optimization problem allows for principled integration of data fitting, landmark alignment, and geometric priors (smoothness).
  • Primal-Dual Methods for Non-Smoothness: The use of primal-dual optimization is key to efficiently solving the non-smooth objective function, enabling real-time performance.
  • Steiner Points Enhance Accuracy: Introducing Steiner points, beyond the sparse VO landmarks, significantly improves the mesh's ability to capture fine geometric details and increases reconstruction accuracy.
  • GPU Acceleration is Critical: Offloading the computationally intensive dual updates to the GPU is essential for achieving the claimed real-time performance.

Key Results

  • Improved Accuracy and Density: The proposed method achieves more accurate and denser 3D mesh reconstructions compared to existing VO pipelines (LSD-SLAM, MLM) and the FLaME method, as evidenced by higher fractions of inverse depths within 10% of ground truth on the EuRoC dataset.
  • Real-time Performance: The algorithm runs in real-time, with an average processing time of 13.2ms per frame (excluding the VIO pipeline's computation).
  • Impact of Steiner Points: Increasing the number of Steiner points leads to noticeable improvements in depth estimation accuracy on the uHumans2 dataset, with diminishing returns beyond a certain density.
  • Smooth and Accurate Meshes: Qualitative results show visually smooth and geometrically accurate mesh reconstructions that faithfully represent the scene's geometry.

Limitations and Future Work

  • Dependency on VO/VIO: The quality of the reconstruction is inherently dependent on the accuracy and density of the visual odometry pipeline's output (sparse landmarks and pose estimates).
  • Single-Frame Reconstruction: The current approach focuses on per-frame reconstruction. The authors acknowledge that fusing these per-frame meshes into a consistent multi-frame scene representation is a promising direction for future work.
  • Computational Cost of Steiner Points: While beneficial, a very high density of Steiner points can push the computation time beyond real-time limits. Future work might explore more adaptive strategies for placing these points.

Summary updated

Abstract

Meshes are commonly used as 3D maps since they encode the topology of the scene while being lightweight. Unfortunately, 3D meshes are mathematically difficult to handle directly because of their combinatorial and discrete nature. Therefore, most approaches generate 3D meshes of a scene after fusing depth data using volumetric or other representations. Nevertheless, volumetric fusion remains computationally expensive both in terms of speed and memory. In this paper, we leapfrog these intermediate representations and build a 3D mesh directly from a depth map and the sparse landmarks triangulated with visual odometry. To this end, we formulate a non-smooth convex optimization problem that we solve using a primal-dual method. Our approach generates a smooth and accurate 3D mesh that substantially improves the state-of-the-art on direct mesh reconstruction while running in real-time.
BibTeX
@inproceedings{rosinol2021smooth,
  author = {Antoni Rosinol and Luca Carlone},
  title = {{Smooth Mesh Estimation from Depth Data using Non-Smooth Convex Optimization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=pTKGyRiXOj},
}
Figure