Real-time Vision-based Depth Reconstruction with NVidia Jetson

Andrey Bokovoy, Kirill Muravyev, Konstantin S. Yakovlev

Introduction

In the field of mobile robotics, particularly for lightweight platforms like micro-aerial vehicles (MAVs), traditional depth sensing solutions such as LiDAR or RGB-D cameras often impose prohibitive costs in terms of weight, power consumption, and price. Monocular depth estimation—the ability to infer 3D structure from a single 2D image—offers a compelling alternative. However, current state-of-the-art deep learning models for this task are typically designed for high-end desktop GPUs, creating a "performance wall" when deployed on embedded hardware.

This research addresses this gap by presenting a framework for real-time, monocular depth estimation optimized for power-constrained embedded systems like the NVidia Jetson TX2. By refining Fully Convolutional Neural Network (FCNN) architectures and introducing an adaptive loss function, the proposed system enables mobile robots to perform visual Simultaneous Localization and Mapping (vSLAM) at rates exceeding 16 FPS using only a standard camera.

Methodology

The core of the approach lies in balancing depth reconstruction accuracy with inference speed. The authors explored several architectural optimizations to maximize efficiency on the Jetson platform.

Architectural Variations

The models utilize a ResNet50-based encoder-decoder structure. To reduce computational overhead, a "Lite Basic" version was developed by pruning the final residual blocks of the encoder. Furthermore, skip connections (shortcuts) were integrated to transfer high-resolution spatial features from the encoder directly to the decoder, which helps preserve sharp object boundaries in the predicted depth maps.

Overview of the FCNN architectural variations, including the 'Lite Basic' pruning and skip connection (SC) configurations.

Efficiency Optimizations

Several key techniques were employed to reach real-time performance:

  • Interleaving for Up-convolution: Traditional unpooling and convolution operations are computationally expensive. The authors implemented an "interleaving" technique that uses four smaller, parallel convolutions. A custom C++/CUDA implementation was developed to execute this in a single operation, significantly reducing memory copy overhead between the CPU and GPU.
  • Factorized Convolutions: Standard $3 \times 3$ convolutions were replaced with "non-bottleneck" blocks (sequences of $3 \times 1$ and $1 \times 3$ convolutions), reducing the number of parameters and operations without a significant loss in representational power.
  • Adaptive BerHu Loss (aBerHu): For robotic navigation, errors in nearby depth estimates are far more critical than errors in distant ones. The authors modified the standard BerHu loss to adaptively shift its threshold during training, forcing the network to prioritize accuracy for obstacles in the immediate vicinity of the robot.

Experimental Results

The models were trained on the NYU Depth Dataset v2. Evaluation on the NVidia Jetson TX2 demonstrated that the optimizations successfully bridged the gap to real-time performance.

Inference Performance

The fastest configuration, "Lite Basic + interl + T," achieved an inference time of approximately 37ms, or ~27 FPS. The most balanced model, "Basic + SC + Upsampling + nonbt," maintained a robust 16 FPS while achieving a Relative Error of 0.173, comparable to much larger, non-real-time models.

System Integration and vSLAM

To demonstrate practical utility, the depth reconstruction node was integrated into the RTAB-Map vSLAM framework. The complete pipeline—encompassing image capture, depth estimation, and 3D mapping—ran at approximately 12 FPS on the Jetson TX2.

The integrated system in action, showing the camera view, predicted refined depth, and the resulting 3D point cloud and Octomap generated in real-time.

As shown in the integrated view, the system successfully generates detailed 3D maps of indoor environments. The detection of loops and subsequent map optimization (as seen in the Octomap status) confirms the stability of the generated depth data for long-term navigation tasks.

Conclusion and Future Work

This work demonstrates that with careful architectural pruning and hardware-aware implementation, high-quality monocular depth estimation is viable for real-time robotics on embedded platforms. While the current 12–16 FPS is sufficient for many tasks, extremely fast robot maneuvers still pose a challenge for visual odometry. Future research will focus on tighter integration between the FCNN and the SLAM backend, potentially using the network's internal feature representations to enhance loop closure detection and pose estimation directly.

Summary updated

BibTeX
@inproceedings{bokovoy2019real,
  author = {Andrey Bokovoy and Kirill Muravyev and Konstantin S. Yakovlev},
  title = {{Real-time Vision-based Depth Reconstruction with NVidia Jetson}},
  booktitle = {ECMR},
  year = {2019},
  doi = {10.1109/ecmr.2019.8870936},
}
Figure