SIUNet: Sparsity Invariant U-Net for Edge-Aware Depth Completion

Avinash Nittur Ramesh, Fabio Giovanneschi, María A. González-HuiciWACV 2023

TL;DR

This paper introduces SIUNet, a novel U-Net architecture for depth completion that is invariant to input sparsity, meaning it can handle very sparse LiDAR data without degradation. By employing an auxiliary depth contour generation task during training, SIUNet explicitly learns to preserve sharp object boundaries, outperforming existing unguided methods in both quantitative metrics and qualitative results, especially in challenging scenarios.

What Problem Does This Paper Address?

Depth completion, the task of generating dense depth maps from sparse measurements, is crucial for applications like autonomous driving and robotics. While traditional methods like stereo cameras provide dense depth, they struggle in low-light or long-range scenarios. LiDAR sensors excel in these conditions but provide sparse point clouds. Existing methods to densify LiDAR data either rely on additional sensor data (like RGB images), which complicates system integration and increases cost, or are unguided but often suffer from "depth bleeding" – blurry object boundaries – especially with extremely sparse inputs. This paper aims to bridge this gap by proposing an unguided depth completion method that is robust to extreme sparsity and explicitly preserves sharp object edges.

Methods

The core of the proposed solution is SIUNet, a sparsity-invariant U-Net architecture. Key methodological contributions include:

  • Sparsity Invariance: The network is designed to handle varying levels of input sparsity without explicit pre-processing or complex data augmentation for sparsity. This is achieved through the use of sparsity invariant convolutional layers adapted from prior work.
  • Auxiliary Depth Contour Learning: To enforce edge-awareness and combat depth bleeding, SIUNet employs an auxiliary learning task. During an initial training phase on synthetic data, the network is trained to predict not only the dense depth map but also a depth contour image. This contour image, derived from semantic segmentation masks, acts as a supervisory signal to explicitly learn structural information and boundary consistency.
  • Two-Stage Training: The network is trained in two stages:
  • Auxiliary Learning (Source Domain): Trained on the SYNTHIA dataset with sparse depth as input and both dense depth and depth contours as targets.
  • Transfer Learning (Target Domain): Fine-tuned on the KITTI dataset using only sparse depth as input and semi-dense depth as the target. This stage leverages the learned features from the first stage.
  • End-to-End Architecture: Unlike some multi-task approaches, SIUNet maintains a simple, single U-Net architecture without branching, facilitating end-to-end feature sharing between the primary depth completion task and the auxiliary contour task. This is possible because both tasks operate in the "depth domain."
  • Novel Loss Function: A combined loss function is used, incorporating Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) for both dense depth and depth contours.

Datasets

  • SYNTHIA [36]: Used as the source domain for the initial auxiliary learning phase. It provides synthetic RGB, depth, and semantic segmentation images.
  • KITTI [13]: Used as the target domain for transfer learning and final evaluation. It provides sparse Velodyne LiDAR depth maps and semi-dense depth ground-truth images.
  • NYU v2 Indoor Dataset [39]: Used for evaluating the model's generalization ability to indoor scenes, demonstrating its robustness beyond the outdoor training domain.

Key Insights and Ideas

  • Depth Contours as a Powerful Auxiliary Signal: Explicitly training for depth contours, derived from semantic segmentation, is a highly effective way to enforce boundary consistency in depth completion. This is more effective than relying solely on dense depth targets, especially for unguided methods.
  • Domain Alignment for Auxiliary Tasks: By using depth contours (which are in the "depth domain") instead of RGB images for the auxiliary task, the network architecture can remain simple and end-to-end, avoiding complex fusion mechanisms and branch-outs.
  • Sparsity Invariance is Crucial for Real-World LiDAR: The paper demonstrates that a network designed to be inherently invariant to input sparsity performs significantly better than standard CNNs when faced with extremely sparse LiDAR data, a common real-world scenario.
  • Transfer Learning from Synthetic to Real is Effective: The two-stage training process, starting with synthetic data and fine-tuning on real-world data, effectively transfers learned features and improves performance on the target domain.

Key Results

  • Quantitative Performance: SIUNet achieved the best MAE (227.28 mm) and iMAE (0.96 km⁻¹) on the KITTI depth completion benchmark among the compared unguided approaches. Its RMSE and iRMSE were also competitive.
  • Qualitative Performance: The reconstructed depth images from SIUNet exhibit significantly sharper object boundaries and less depth bleeding compared to other state-of-the-art unguided methods, even for extremely sparse inputs (e.g., 95% sparsity).
  • Sparsity Robustness: The model maintains good performance and sharp edges even with input sparsity levels up to 95%, reconstructing approximately 270,000 pixels from only ~1,000 valid depth pixels with an MAE of 512 mm.
  • Generalization: SIUNet showed competitive results on the indoor NYU v2 dataset, despite being trained on outdoor data, highlighting its generalization capabilities.

Limitations and Future Work

  • Limitations:
    • The auxiliary depth contour generation is performed only on the source domain (SYNTHIA). While effective, this means the contour generation step is not directly applied during transfer learning on the target domain.
  • The paper focuses on unguided depth completion, meaning it doesn't leverage RGB or semantic information from the target domain during inference.
  • Future Work:
    • The authors suggest that their approach could be extended to scenarios where real-world ground-truth images and other sensor data are unavailable by opting for a zero-shot learning approach, leveraging the model's robustness to sparsity and patterns.
  • Further exploration of the generalization ability to different sensor systems and environments is implied by the NYU v2 experiment.

Summary updated

BibTeX
@inproceedings{ramesh2023siunet,
  author = {Avinash Nittur Ramesh and Fabio Giovanneschi and María A. González-Huici},
  title = {{SIUNet: Sparsity Invariant U-Net for Edge-Aware Depth Completion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure