Recognition and Depth Estimation of Ships Based on Binocular Stereo Vision

Yuanzhou Zheng, Peng Liu, Long Qian, Shiquan Qin, Xinyu Liu, Yong Ma, Ganjun ChengJournal of Marine Science and Engineering 2022 · DOI

TL;DR

This paper presents a novel system for inland river ships that combines efficient ship recognition with accurate depth estimation using binocular stereo vision. By adapting the YOLOv4 architecture with MobileNetV1 and employing a sub-pixel feature matching technique with FSRCNN, the system achieves high accuracy while significantly reducing computational load, paving the way for safer and more intelligent maritime navigation.

What Problem Does This Paper Address?

Maritime transportation is crucial for global trade, but the increasing volume of ships, especially on inland waterways, leads to a rise in traffic accidents. Existing safety measures like Vessel Traffic Service (VTS) and Automatic Identification System (AIS) have limitations, particularly in complex inland environments and when AIS data is unreliable. The core challenge is to develop robust and efficient environmental perception systems that can accurately identify ships and determine their distances, thereby enhancing navigation safety and supporting the development of intelligent ships.

Methods

The proposed system tackles this challenge in two main stages:

  • Ship Recognition: To address the computational burden of the standard YOLOv4 model, the authors replace its CSPDarknet53 backbone with the lightweight MobileNetV1. This creates the MobileNetV1-YOLOv4 model, designed to maintain high recognition accuracy while drastically reducing model size and computational requirements.
  • Depth Estimation: This stage involves a multi-step process:
  • Super-resolution Reconstruction: The FSRCNN algorithm is used to enhance the resolution of the input stereo images. This increases the density and accuracy of feature points, which is crucial for precise parallax calculation.
  • Feature Point Detection and Matching: The ORB algorithm is employed for detecting and matching feature points at a sub-pixel level. This robust algorithm is chosen for its speed and invariance to noise and rotation.
  • Depth Calculation: Using the matched feature points and the principles of binocular stereo vision (triangulation), the system calculates the depth (distance) of the detected ships.

Datasets

  • Ship Image Collection: The authors collected 2000 images of inland river ships (container ships, passenger ships, and ore carriers) using a binocular stereo camera.
  • Training Data:
    • For the FSRCNN network: ImageNet-91 dataset.
  • For the MobileNetV1-YOLOv4 model: A dataset comprising randomly cropped and stitched ship images from multiple angles (likely derived from their collected data and potentially augmented). The dataset was split into training (90%) and testing (10%) sets.
  • Testing Data: Set14 dataset was used for testing the FSRCNN network.

Key Insights and Ideas

  • Lightweight Backbone for Object Detection: Replacing the heavy CSPDarknet53 backbone in YOLOv4 with MobileNetV1 is a highly effective strategy for reducing computational load without significantly sacrificing recognition accuracy, making it suitable for resource-constrained maritime applications.
  • Super-Resolution for Enhanced Feature Matching: Using FSRCNN to upsample images before feature detection is a clever way to overcome the sparsity of features in standard images, leading to more reliable parallax estimation and thus more accurate depth measurements.
  • Sub-pixel Feature Matching is Crucial: The paper emphasizes the importance of sub-pixel level feature detection and matching (via ORB) for achieving precise depth estimation, especially in challenging environments.
  • Integrated System for Safety: The combination of an efficient recognition system and an accurate depth estimation system addresses a critical gap in current maritime perception technologies, directly contributing to improved navigation safety.

Key Results

  • Ship Recognition: The MobileNetV1-YOLOv4 model achieved an mAP of 89.25%. The backbone network's weight size was reduced to 47.6 M, approximately 20% of the classic YOLOv4. The FPS (frames per second) increased significantly to 66.23 compared to YOLOv4's 26.11.
  • Depth Estimation: When the target distance is around 300m, the depth estimation error is less than 3%. For distances up to 1000m, the error is within ±5%. The system demonstrated good performance across different ship types.
  • FSRCNN Performance: FSRCNN reconstruction resulted in higher PSNR values compared to ESPCN, indicating better image quality and less distortion.

Limitations and Future Work

  • Recognition Accuracy in Simple Scenes: The authors acknowledge that the recognition performance might be indirectly affected by the relatively simple inland river scenes and the limited variety of ship types in their dataset.
  • Environmental Factors: The ORB algorithm's performance in depth estimation can be affected by environmental factors like weather and illumination, which can degrade image quality and feature detection.
  • Future Work: The authors suggest that further research will focus on addressing these limitations. This likely includes expanding the dataset with more diverse scenarios and ship types, and exploring more robust feature detection and matching techniques that are less sensitive to environmental variations.

Summary updated

Abstract

To improve the navigation safety of inland river ships and enrich the methods of environmental perception, this paper studies the recognition and depth estimation of inland river ships based on binocular stereo vision (BSV). In the stage of ship recognition, considering the computational pressure brought by the huge network parameters of the classic YOLOv4 model, the MobileNetV1 network was proposed as the feature extraction module of the YOLOv4 model. The results indicate that the mAP value of the MobileNetV1-YOLOv4 model reaches 89.25%, the weight size of the backbone network was only 47.6 M, which greatly reduced the amount of computation while ensuring the recognition accuracy. In the stage of depth estimation, this paper proposes a feature point detection and matching algorithm based on the ORB algorithm at sub-pixel level, that is, firstly, the FSRCNN algorithm was used to perform super-resolution reconstruction of the original image, to further increase the density of image feature points and detection accuracy, which was more conducive to the calculation of the image parallax value. The ships’ depth estimation results indicate that when the distance to the target is about 300 m, the depth estimation error is less than 3%, which meets the depth estimation needs of inland ships. The ship target recognition and depth estimation technology based on BSV proposed in this paper makes up for the shortcomings of the existing environmental perception methods, improves the navigation safety of ships to a certain extent, and greatly promotes the development of intelligent ships in the future.
BibTeX
@article{zheng2022recognition,
  author = {Yuanzhou Zheng and Peng Liu and Long Qian and Shiquan Qin and Xinyu Liu and Yong Ma and Ganjun Cheng},
  title = {{Recognition and Depth Estimation of Ships Based on Binocular Stereo Vision}},
  journal = {Journal of Marine Science and Engineering},
  year = {2022},
  doi = {10.3390/jmse10081153},
}
Figure