Multi-Scale Object Detection Model for Autonomous Ship Navigation in Maritime Environment

Zeyuan Shao, Hongguang Lyu, Yong Yin, Tao Cheng, Xiaowei Gao, Wenjun Zhang, Qianfeng Jing, Yanjie Zhao, Lunping ZhangJournal of Marine Science and Engineering 2022 · DOI

TL;DR

This paper introduces VarifocalNet*, an enhanced convolutional neural network designed for robust multi-scale object detection in challenging maritime environments. By incorporating deformable convolutions, a new loss function, and soft non-maximum suppression, the model significantly improves the detection of small and overlapping objects, crucial for autonomous ship navigation.

What Problem Does This Paper Address?

Autonomous ships rely heavily on accurate and reliable sensing to navigate safely. While marine radar is a standard tool, it struggles with blind spots, weak radar signals, and small objects. Electro-optical (EO) sensors, like cameras, offer richer visual information but face significant challenges in maritime settings: waves and reflections distort images, varying observation angles alter object appearances, and complex backgrounds (like ports) can obscure targets. Existing object detection models often perform well on large objects but falter when detecting smaller ones, which are critical for collision avoidance. This paper aims to bridge this gap by developing a robust multi-scale object detection system specifically for these harsh and dynamic maritime conditions.

Methods

The core of this work is an enhanced version of the VarifocalNet model, dubbed VarifocalNet*. The enhancements focus on improving feature extraction, detection accuracy, and robustness:

  • Deformable Convolutional Networks (DCN): Traditional convolutions use fixed-size receptive fields. DCNs introduce learnable offsets to the sampling grid, allowing the network to adapt to geometric transformations and extract features more effectively, especially for objects with varying shapes or orientations due to camera angles.
  • DIoU Loss Function: The paper replaces the standard IoU loss with the Distance-IoU (DIoU) loss. DIoU considers not only the overlap between predicted and ground truth bounding boxes but also the distance between their center points. This leads to faster convergence and more accurate localization.
  • Soft Non-Maximum Suppression (SNMS): To address issues with overlapping objects, especially in dense scenes like ports, the authors employ an improved SNMS algorithm with Gaussian weighting. This method is more robust than traditional NMS, which can lead to missed detections in crowded scenarios.
  • Multi-Scale Prediction: The model leverages feature pyramid networks (FPN) and an image pyramid approach during training. This allows the network to capture features at various scales, improving the detection of both small and large objects.

The backbone of the network is ResNet50, enhanced with DCNs in its later stages, and FPN is used for feature map processing.

Datasets

The primary dataset used for training and evaluation is the Marine Obstacle Detection Dataset v2.0 (MODD2). This dataset was collected by a USV in the port of Koper, Slovenia, and is characterized by:

  • Strong sunlight and water reflections.
  • Dense targets near ports with overlapping and occluded objects.
  • Large variations in object sizes.
  • Numerous small objects near the horizon.

The authors re-labeled and counted objects in MODD2 according to the Microsoft COCO dataset's definition of large (pixel area > 96²), medium (32² < pixel area < 96²), and small objects (pixel area < 32²). The dataset was split into 6225 training images and 691 test images. Data augmentation techniques like random flipping, brightness adjustment, and rotation were applied.

The paper also mentions testing the generalization performance on another water surface challenge dataset (dataset from reference [49]), which includes dense small fishing boats, sea fog, dawn scenes, and night scenes.

Key Insights and Ideas

  • Maritime Environment Specifics: The paper highlights that maritime environments are uniquely challenging due to dynamic conditions (waves, reflections, fog) that standard computer vision algorithms struggle with. Tailoring models to these specific conditions is crucial.
  • Small Object Detection is Key: A recurring theme is the difficulty and importance of detecting small objects for autonomous navigation. The proposed enhancements specifically target this weakness.
  • DCN for Geometric Adaptability: The introduction of DCNs is a significant step in making object detection models more resilient to the varied orientations and shapes of objects encountered at sea.
  • DIoU for Better Localization: Moving beyond simple overlap, incorporating center point distance in the loss function (DIoU) directly improves bounding box accuracy.
  • SNMS for Dense Scenes: The maritime environment, especially near ports, often presents dense clusters of objects. SNMS proves essential for preventing missed detections in such scenarios.

Key Results

The proposed VarifocalNet* model demonstrates significant improvements over baseline and competing methods:

  • Overall Accuracy: The model achieves an average precision rate of 78.9% on the MODD2 dataset, with a precision of 98.6% at an IoU threshold of 0.5.
  • Small Object Performance: VarifocalNet* shows a substantial improvement in detecting small objects, with a 7.2% increase in precision compared to the baseline VarifocalNet. The recall rate for small objects also increased by 12.8%.
  • Comparison to State-of-the-Art: When compared to SSD, YOLOv3, Faster R-CNN, RetinaNet, and Cascade R-CNN, VarifocalNet* consistently outperforms them, particularly in terms of precision for small objects (e.g., 31% improvement over Faster R-CNN).
  • Robustness: Visual comparisons in Figures 12-14 demonstrate VarifocalNet*'s superior performance in challenging conditions like dense occlusion, strong sunlight, and sea fog, with fewer missed detections and false positives.
  • Real-time Performance: While DCN adds a small number of parameters and multi-scale prediction increases inference time, the overall impact on real-time performance is considered acceptable for engineering applications, with inference times of 91ms per image in the fully enhanced model.

Limitations and Future Work

  • Hyperparameter Tuning: The authors acknowledge that further research is needed to fine-tune CNN hyperparameters for optimal performance.
  • Data Dependency: The high precision of VarifocalNet* relies on a large amount of labeled data. The cost of manual annotation is high, and adaptability to new scenarios is limited.
  • Future Directions:
    • Engineering Applications: Converting the model to TensorRT format and deploying it on autonomous ships.
  • Model Complexity Reduction: Reducing model complexity to further enhance real-time performance.
  • Spatiotemporal Data: Investigating the use of video data (spatiotemporal information) to reduce false detection rates, as successive frames exhibit temporal dependence.
  • Weak/Unsupervised Learning: Exploring weak or unsupervised learning methods, potentially using Generative Adversarial Networks (GANs), to augment maritime datasets and improve adaptability.

Summary updated

Abstract

Accurate detection of sea-surface objects is vital for the safe navigation of autonomous ships. With the continuous development of artificial intelligence, electro-optical (EO) sensors such as video cameras are used to supplement marine radar to improve the detection of objects that produce weak radar signals and small sizes. In this study, we propose an enhanced convolutional neural network (CNN) named VarifocalNet * that improves object detection in harsh maritime environments. Specifically, the feature representation and learning ability of the VarifocalNet model are improved by using a deformable convolution module, redesigning the loss function, introducing a soft non-maximum suppression algorithm, and incorporating multi-scale prediction methods. These strategies improve the accuracy and reliability of our CNN-based detection results under complex sea conditions, such as in turbulent waves, sea fog, and water reflection. Experimental results under different maritime conditions show that our method significantly outperforms similar methods (such as SSD, YOLOv3, RetinaNet, Faster R-CNN, Cascade R-CNN) in terms of the detection accuracy and robustness for small objects. The maritime obstacle detection results were obtained under harsh imaging conditions to demonstrate the performance of our network model.
BibTeX
@article{shao2022multi,
  author = {Zeyuan Shao and Hongguang Lyu and Yong Yin and Tao Cheng and Xiaowei Gao and Wenjun Zhang and Qianfeng Jing and Yanjie Zhao and Lunping Zhang},
  title = {{Multi-Scale Object Detection Model for Autonomous Ship Navigation in Maritime Environment}},
  journal = {Journal of Marine Science and Engineering},
  year = {2022},
  doi = {10.3390/jmse10111783},
}
Figure