RealNet: Combining Optimized Object Detection with Information Fusion Depth Estimation Co-Design Method on IoT

Zhuohao Li, Fandi Gou, Qixin De, Leqi Ding, Yuanhang Zhang, Yunze CaiarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces RealNet, a novel co-designed system for real-time object detection and depth estimation on resource-constrained IoT devices like the Jetson Nano. By optimizing YOLOv5 for speed and fusing its outputs with a novel Virtual Normal Loss (VNL) based depth estimation method, RealNet achieves significant improvements in both accuracy and processing speed for autonomous navigation tasks.

What Problem Does This Paper Address?

Object detection and depth estimation are crucial for applications like autonomous driving, augmented reality, and robotics. However, achieving high accuracy and real-time performance simultaneously, especially on embedded systems with limited computational power (like IoT devices), remains a significant challenge. Existing methods often involve a trade-off between speed and accuracy, or require expensive, specialized hardware. This paper aims to bridge this gap by developing an integrated system that optimizes both tasks for efficient deployment on platforms like the Jetson Nano, enabling real-time perception for mobile robots.

Methods

The core of RealNet is a co-design approach that integrates two key components:

  • Optimized Object Detection (YOLOv5): The authors leverage the YOLOv5 architecture, known for its speed and accuracy. They specifically focus on optimizing its DarkNet53 backbone and simplifying the network structure to achieve a prediction speed of as low as 0.01s per frame. This optimization is crucial for real-time applications.
  • Information Fusion Depth Estimation:
    • VNL Depth Estimation: This method is based on a novel "Virtual Normal Loss" (VNL) that considers geometric constraints in 3D space. Instead of solely relying on pixel-wise depth, VNL constructs virtual normal vectors from estimated depth maps and uses them to supervise the depth estimation process. This approach aims to improve robustness and capture richer depth information.
  • Perspective-n-Point (PnP) for Motion Estimation: To address the potentially lower frame rate of the VNL output and provide a faster depth estimation for motion, the authors incorporate a Perspective-n-Point (PnP) algorithm. This method uses feature matching (Shi-Tomasi corners and Lucas-Kanade optical flow) to estimate the relative pose change between consecutive frames, providing a quick depth estimate.
  • Information Fusion: The VNL output and the PnP-derived depth information are then fused. This fusion aims to compensate for the limitations of each individual method, particularly the insufficient frame rate of VNL and the potential scale ambiguity of PnP. A Kalman filter is employed to smooth the fused depth information and eliminate jitter, leading to more stable and accurate depth predictions.
  • Control System: The system integrates target detection results and fused depth information to calculate the target's position. A pure pursuit control algorithm is then used to track the target.

Datasets

Not explicitly detailed in the paper. The authors mention collecting data by controlling a moving target and shooting videos, then using these for training. They also refer to using "a variety of data enhancement methods" and a "pre-training model" (yolov5_s.pt), implying the use of standard datasets for initial training, but specific dataset names are not provided.

Key Insights and Ideas

  • Co-design is Key: The paper highlights that optimizing object detection and depth estimation independently might not yield the best real-time performance on embedded systems. A co-designed approach, where the strengths of each task are leveraged and weaknesses are compensated through fusion, is more effective.
  • Virtual Normal Loss (VNL) for Robust Depth: The VNL method offers a promising direction for monocular depth estimation by incorporating global geometric constraints, leading to more robust and informative depth predictions beyond simple pixel-wise values.
  • PnP as a Fast Complement: Using PnP with feature matching provides a computationally efficient way to estimate depth and motion, especially when high frame rates are critical, complementing slower but potentially more accurate deep learning methods.
  • Kalman Filter for Stability: The application of a Kalman filter to fuse and smooth depth estimates is crucial for achieving stable and reliable depth information, essential for control applications.
  • YOLOv5 Optimization for IoT: Demonstrating that significant speedups can be achieved on YOLOv5 for deployment on resource-constrained devices like the Jetson Nano is a practical contribution.

Key Results

  • Object Detection Speed: The optimized YOLOv5 achieves a prediction speed of 0.01s per frame.
  • Depth Estimation Improvement: The proposed information fusion method (VNL + PnP + Kalman Filter) is reported to be 3 times faster than pure depth estimation neural network methods.
  • Accuracy Improvement: The accuracy of the depth estimation is improved by 13% compared to the original VNL method.
  • Robustness: The method effectively eliminates jitter in depth information and improves robustness.
  • Deployment: The system was successfully deployed on a Jetson-Nano platform and controlled a trolley for target tracking.

Limitations and Future Work

  • Dataset Specificity: The paper mentions data collection through video recording but does not specify standard datasets used for training or evaluation, which limits reproducibility and direct comparison with other works.
  • Scale Ambiguity: While PnP is used, the paper acknowledges the inherent scale ambiguity of monocular vision, which might still affect absolute distance measurements in certain scenarios.
  • Real-world Complexity: The experiments were conducted with controlled moving targets. Performance in more complex, dynamic, and cluttered real-world environments might require further validation.
  • Future Work: The authors mention releasing new versions of the system, suggesting ongoing development and potential improvements in performance and functionality. They also hint at exploring more advanced fusion strategies and control algorithms.

Summary updated

BibTeX
@misc{li2022realnet,
  author = {Zhuohao Li and Fandi Gou and Qixin De and Leqi Ding and Yuanhang Zhang and Yunze Cai},
  title = {{RealNet: Combining Optimized Object Detection with Information Fusion Depth Estimation Co-Design Method on IoT}},
  year = {2022},
  eprint = {2204.11216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.11216},
}
Figure