Lightweight Monocular Depth Estimation Model by Joint End-to-End Filter pruning

Sara Elkerdawy, Hong Zhang, Nilanjan Ray

Abstract

Convolutional neural networks (CNNs) have emerged as the state-of-the-art in multiple vision tasks including depth estimation. However, memory and computing power requirements remain as challenges to be tackled in these models. Monocular depth estimation has significant use in robotics and virtual reality that requires deployment on low-end devices. Training a small model from scratch results in a significant drop in accuracy and it does not benefit from pre-trained large models. Motivated by the literature of model pruning, we propose a lightweight monocular depth model obtained from a large trained model. This is achieved by removing the least important features with a novel joint end-to-end filter pruning. We propose to learn a binary mask for each filter to decide whether to drop the filter or not. These masks are trained jointly to exploit relations between filters at different layers as well as redundancy within the same layer. We show that we can achieve around 5x compression rate with small drop in accuracy on the KITTI driving dataset. We also show that masking can improve accuracy over the baseline with fewer parameters, even without enforcing compression loss.

Introduction

Monocular depth estimation—the ability to predict 3D distance from a single 2D image—is a critical task for robotics, augmented reality, and autonomous driving. While CNNs have mastered this task, they are typically massive and power-hungry, requiring high-end GPUs for real-time performance. Simply designing a small model from the start usually leads to poor accuracy because these models miss out on the rich feature representations learned by over-parameterized larger networks. Traditional pruning methods often require tedious, manual layer-by-layer tuning. This paper proposes an automated, joint pruning strategy to transform a large model into a lean, efficient one without sacrificing the intelligence gained during initial training.

Methodology

The authors propose a joint end-to-end filter pruning approach that integrates the pruning process directly into the training loop rather than performing it as a post-processing step.

Binary Masking and Optimization

For every convolutional filter in the network, a real-valued mask is introduced. This mask passes through a Sigmoid function and a binarization step (thresholding at 0.5) to create a "gate." If the gate is 1, the filter is retained; if 0, it is dropped. To handle the non-differentiable nature of binarization during backpropagation, the authors utilize a Straight-Through Estimator (STE) to update the real-valued masks.

The proposed joint end-to-end filter pruning mechanism using binary masks.

The network is trained on two simultaneous goals: a Task Loss for unsupervised depth estimation accuracy and a Sparsity Loss (an $\ell_1$-norm penalty) that rewards the network for deactivating as many filters as possible. Unlike sequential methods, this approach optimizes all masks across all layers simultaneously, allowing the network to exploit global redundancies.

Datasets

The model was evaluated on the KITTI Dataset (using both the Eigen and 2015 stereo splits) and the Cityscapes dataset. Cityscapes was used for pre-training to provide the model with a robust feature base before fine-tuning on the specific driving scenes of KITTI.

Results and Discussion

The results confirm that "starting big and shrinking" allows the model to retain complex spatial relationships that small models trained from scratch fail to capture.

Quantitative Performance

The method achieved a ~5x compression rate, reducing parameters from approximately 30.7M to 5.9M in VGG-based architectures. On the KITTI dataset, this 80% reduction in size resulted in only a marginal 1.9% drop in accuracy.

Quantitative comparison on the KITTI dataset showing compression rates and accuracy metrics.

As shown in the performance table, the pruned models ($VGG + L_{total}$) outperform other lightweight models like PyD-Net across various error metrics (Abs Rel, RMS) while maintaining a significantly smaller footprint than the original VGG baseline. Interestingly, the masking mechanism also acts as a form of regularization; even without enforcing compression loss, it can improve accuracy over the baseline.

Qualitative Insights and Limitations

Qualitatively, the pruned model produces smoother and more detailed depth maps than competing lightweight architectures. However, the authors noted that the pruning process can struggle with very fine details. Small, thin objects like distant poles are often the first features lost when filters are removed to meet compression targets.

Conclusion

Joint end-to-end filter pruning provides an efficient way to deploy high-performance monocular depth estimation on resource-constrained devices. By learning to prune filters globally and jointly, the system identifies redundancies that layer-by-layer methods miss. Future work may explore knowledge distillation to help the pruned models recover fine-grained details lost during the compression process.

Summary updated

Abstract

Convolutional neural networks (CNNs) have emerged as the state-of-the-art in multiple vision tasks including depth estimation. However, memory and computing power requirements remain as challenges to be tackled in these models. Monocular depth estimation has significant use in robotics and virtual reality that requires deployment on low-end devices. Training a small model from scratch results in a significant drop in accuracy and it does not benefit from pre-trained large models. Motivated by the literature of model pruning, we propose a lightweight monocular depth model obtained from a large trained model. This is achieved by removing the least important features with a novel joint end-to-end filter pruning. We propose to learn a binary mask for each filter to decide whether to drop the filter or not. These masks are trained jointly to exploit relations between filters at different layers as well as redundancy within the same layer. We show that we can achieve around 5x compression rate with small drop in accuracy on the KITTI driving dataset. We also show that masking can improve accuracy over the baseline with fewer parameters, even without enforcing compression loss.
BibTeX
@inproceedings{elkerdawy2019lightweight,
  author = {Sara Elkerdawy and Hong Zhang and Nilanjan Ray},
  title = {{Lightweight Monocular Depth Estimation Model by Joint End-to-End Filter pruning}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=JfL5pX-C9Zg},
}
Figure