Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural Regularities From Visual SLAM

Jinwoo Jeon, Hyunjun Lim, Dong-Uk Seo, Hyun MyungIEEE RA-L 2022

Introduction

Visual Simultaneous Localization and Mapping (SLAM) serves as the primary sensory mechanism for robotic navigation. However, most efficient SLAM systems are feature-based, calculating depth only for a limited set of points such as corners or edges. This results in highly sparse depth maps that are often insufficient for complex tasks like obstacle avoidance or autonomous path planning.

While deep learning models have been developed to "complete" these sparse maps, they frequently rely on point features that are sensitive to lighting changes and fail to capture the underlying structural geometry of a scene. Furthermore, many unsupervised models suffer from a "copy-paste" effect, where they blur or replicate nearby pixels rather than understanding the scene's physical layout. This paper introduces Struct-MDC, a framework designed to bridge the gap between traditional geometric SLAM and deep learning by utilizing structural regularities to generate dense, accurate depth maps without the need for ground-truth labels.

Methodology

The Struct-MDC framework operates through a three-step pipeline that integrates classical geometry with neural refinement:

  1. Structural Sketch Generation: Moving beyond simple point clouds, the system extracts line features from the SLAM front-end. It utilizes Constrained Delaunay Triangulation (CDT) to establish a 2D mesh where these lines act as fixed boundaries. By interpolating depth within this mesh, the system creates a structurally sound, albeit coarse, initial depth map.
  2. Mesh Depth Refinement (MDR): The coarse mesh typically lacks fine detail and is limited to the area covered by detected features (the convex hull). The MDR module is a neural network that takes the RGB image, the coarse mesh, and a binary mask as inputs. It "transfers" high-frequency details from the image into the depth map and learns to predict depth for regions outside the initial geometric mesh.
  3. Final Depth Estimation: The refined mesh is passed into a main depth estimation module based on the KBNet architecture. Because the input is already a high-quality structural prior rather than a sparse scattering of dots, the network can produce a significantly more accurate final dense depth map.

The entire system is trained in an unsupervised manner, learning through photometric consistency—comparing reconstructed images from different camera frames—rather than relying on labeled depth data.

Experimental Setup and Datasets

The researchers evaluated Struct-MDC using three distinct benchmarks to test its versatility and robustness:

  • VOID: A standard benchmark for unsupervised depth completion in diverse environments.
  • NYUv2: A popular indoor dataset, re-sampled to simulate the specific sparsity patterns encountered in real-world SLAM.
  • PLAD (Point Line feature and Depth): A new custom dataset introduced by the authors. Collected using an Azure Kinect and IMU, PLAD provides high-quality line and point features from a real SLAM system (UV-SLAM), highlighting the specific challenges of non-uniform feature distribution in robotics.

Key Results and Insights

Struct-MDC demonstrates that incorporating geometric priors significantly improves depth accuracy and visual fidelity:

  • State-of-the-Art Performance: Struct-MDC outperformed leading unsupervised methods like VOICED and FusionNet. On the VOID dataset, it achieved a Mean Absolute Error (MAE) of 111.3mm, a substantial improvement over the 143.8mm baseline.
  • Competitive with Supervised Learning: Remarkably, on the NYUv2 dataset, the unsupervised Struct-MDC achieved a lower MAE than CSPN, a well-known supervised method.
  • Structural Integrity: Qualitative results indicate that Struct-MDC excels at capturing sharp object boundaries, such as chair legs and doorways. By forcing the mesh to respect line features, the model avoids the "bleeding" artifacts and jittery edges common in point-only completion methods.
  • The Role of the Mask: By providing the network with a binary mask of existing SLAM features, the MDR module effectively distinguishes between confirmed geometric data and predicted depth, leading to superior performance at object boundaries.

Conclusion and Future Work

Struct-MDC proves that deep learning models do not need to learn physics from scratch; instead, they can be significantly enhanced by providing a "structural sketch" derived from classical geometry. The primary trade-off for this precision is computational overhead. Struct-MDC processes frames in approximately 32.19 ms (~31 Hz), which is roughly twice the processing time of simpler baselines. While this is sufficient for real-time operation on many platforms, future work will focus on model optimization to reduce latency for low-power mobile robotics while maintaining high structural integrity.

Summary updated

BibTeX
@article{jeon2022struct,
  author = {Jinwoo Jeon and Hyunjun Lim and Dong-Uk Seo and Hyun Myung},
  title = {{Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural Regularities From Visual SLAM}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
}
Figure