TL;DR
This paper introduces L-ENet, a novel lightweight neural network for fast and accurate depth estimation from a single image. By employing ordinal regression and efficient convolutional techniques, L-ENet significantly reduces model size and computational cost, making it suitable for real-time applications and embedded systems.
What Problem Does This Paper Address?
Estimating the depth of objects and scenes is a fundamental problem in computer vision with applications ranging from 3D modeling to autonomous driving. Traditional methods often rely on multiple images (stereo vision) or specialized hardware like infrared sensors or LiDAR, which can be expensive, computationally intensive, or impractical for real-time use. While single-image depth estimation using deep learning has emerged as a promising alternative, existing methods often require large, computationally demanding models, limiting their deployment on resource-constrained devices. This paper aims to bridge this gap by developing a lightweight yet effective neural network capable of fast and accurate depth estimation from a single image.
Methods
The core of the proposed method is a Lightweight Efficient Neural Network (L-ENet), designed to be computationally efficient while maintaining high accuracy. Key methodological contributions include:
- Ordinal Regression for Depth: Instead of treating depth as a continuous regression problem (which can lead to larger errors for distant objects), L-ENet classifies depth into discrete ranges using ordinal regression. This approach mimics human perception, which often estimates depth ranges rather than exact values. A "spacing-increasing discretization" (SID) strategy is used to discretize depth intervals in log space, allowing for more accurate estimation of smaller depths and rational estimation of larger depths.
- Lightweight Architecture: L-ENet is based on the ENet architecture but significantly modified for depth estimation. It removes the decoder part of ENet and replaces general convolutions with depth-wise separable convolutions. This technique separates spatial and channel-wise convolutions, drastically reducing the number of parameters and computations.
- Efficient Convolutional Techniques: The network incorporates atrous (dilated) convolutions and asymmetric convolutions to increase the receptive field without increasing the number of parameters. Atrous spatial pyramid pooling (ASPP) is also used to capture features at multiple scales, improving robustness to object size variations.
- Parametric ReLU (PReLU): To address the dying ReLU problem, L-ENet uses PReLU as its activation function, which allows for a small, trainable gradient for negative inputs, leading to more flexible training.
Datasets
The proposed L-ENet was trained and evaluated on two widely used datasets:
- NYU Depth v2: This dataset comprises indoor scenes captured by a Microsoft Kinect, providing RGB images and corresponding depth maps.
- KITTI Dataset: This dataset consists of outdoor scenes captured from moving vehicles, including RGB images and LiDAR-derived depth information.
Key Insights and Ideas
- Ordinal Regression Outperforms Continuous Regression for Distant Objects: The paper highlights that treating depth as a continuous regression problem leads to significant errors for distant objects. Ordinal regression, by classifying depth into ranges, effectively mitigates this issue.
- Lightweight Architectures are Crucial for Real-Time Applications: The significant reduction in model size and parameters achieved by L-ENet is key to enabling depth estimation on embedded systems and smartphones, where computational resources are limited.
- Depth-wise Separable Convolutions are Highly Effective: This technique is central to L-ENet's efficiency, demonstrating a substantial decrease in parameters and computational cost compared to standard convolutions without sacrificing performance.
- ASPP Enhances Multi-Scale Feature Extraction: The inclusion of ASPP allows the network to effectively process objects at various scales within a single image, leading to more accurate depth predictions.
Key Results
- NYU Depth v2: L-ENet achieved competitive results, outperforming several deep neural network-based approaches. While slightly outperformed by DORN [14] on some metrics, L-ENet offers significantly fewer parameters and faster computation, making it more suitable for real-time systems.
- KITTI Dataset: L-ENet demonstrated strong performance, showing improvements over DORN [14] and SORD [27] on key metrics like $\delta < 1.25$ and RMSElog.
- Model Compression: L-ENet's model size was reduced by approximately 12.7 times, and the number of parameters by 12.25 times compared to DORN [14]. Processing time was reduced from 21.6 seconds to 4.3 seconds on a CPU.
- Robustness: The model showed reasonable robustness to added noise (Gaussian, Salt & Pepper, Poisson) on the KITTI dataset.
- Qualitative Results: Visualizations show that L-ENet accurately estimates depth maps, even in complex backgrounds, and correctly captures coarse variations between objects.
Limitations and Future Work
- Upsampling Layer: The paper notes that for dense and accurate depth prediction, especially for near indoor data, adjusting the steps of the upsampling layer might be necessary. The current reduction to a single step for speed can lead to larger prediction errors in certain scenarios.
- Future Work: The authors suggest extending L-ENet to capture finer details by including convex, concave, and occlusion edge labels. They also aim to further simplify the network by reducing the number of layers and test its application on smartphones and embedded systems. The paper also mentions the potential for readers to find application videos demonstrating its use in intelligent vehicle depth map extraction and 3D modeling.
Summary updated