Application of Convolutional Neural Network (CNN) to Recognize Ship Structures

Jae-Jun Lim, Dae‐Won Kim, Woon-Hee Hong, Min Gyu Kim, Dong-Hun Lee, Sun Young Kim, Jae Hoon JeongSensors 2022 · DOI

TL;DR

This paper introduces a novel approach to enhance the safety of drone-based maritime delivery services by developing a Convolutional Neural Network (CNN) system capable of accurately recognizing ship structures. By creating a new dataset of drone-captured ship imagery and leveraging the Detectron2 platform, the researchers demonstrate improved ship and structure identification, crucial for autonomous navigation and collision avoidance.

What Problem Does This Paper Address?

The burgeoning field of drone-based maritime delivery, particularly for services like shore-to-ship deliveries, faces a critical challenge: ensuring safe and reliable autonomous navigation. While Automatic Identification Systems (AIS) provide ship location and basic information, they lack the visual detail necessary for precise navigation and collision avoidance. This means drones cannot reliably "see" and understand the complex structures of ships, which can pose risks during flight and landing. Traditional methods of manual verification are time-consuming and require expert knowledge. This paper aims to bridge this gap by developing an AI system that can visually identify ships and their critical structures from drone-captured imagery, thereby enhancing the safety and efficiency of these emerging delivery services.

Methods

The core of this research lies in applying Convolutional Neural Networks (CNNs) for object detection and segmentation. Specifically, the authors utilize the Detectron2 platform, a powerful framework for object detection and segmentation research. They compare two prominent CNN architectures: Faster R-CNN and Mask R-CNN.

  • Faster R-CNN is used as a baseline for object detection, focusing on bounding box classification.
  • Mask R-CNN is employed for instance segmentation, which goes a step further by not only detecting objects but also generating a pixel-level mask for each instance. This is crucial for understanding the precise shape and boundaries of ship structures.

The researchers experimented with different backbone networks (R50-C4, R50-DC5, R50-FPN) within these architectures to find the optimal configuration for ship structure recognition. They also highlight the importance of the RoIAlign layer over RoIPool for more accurate segmentation.

Datasets

The paper introduces a new dataset called the Marine Traffic Management Net (MTMnet) dataset. This dataset is unique because it consists of images captured by drones during actual maritime delivery operations in Busan Port.

  • MTMnet Dataset Characteristics:
    • Images are taken from aerial perspectives, reflecting a drone's viewpoint.
  • It includes up-to-date visual information on ship appearances, including painting and structures.
  • Images are captured from various angles and distances, providing diverse visual cues.
  • The dataset was created by annotating approximately 1536 image files with polygonal annotations, which were then converted to the COCO dataset format.

This dataset is contrasted with existing public datasets (e.g., MODD, SMD, IPATCH) which often rely on satellite imagery or web searches and may not accurately represent the visual challenges faced by drones.

Key Insights and Ideas

  • Visual Data is Crucial for Drone Navigation: The study strongly emphasizes that while AIS provides location data, it's insufficient for safe drone operations. Accurate visual recognition of ship structures is paramount for collision avoidance and precise maneuvering.
  • Mask R-CNN Outperforms Faster R-CNN for Structure Recognition: The research clearly demonstrates that Mask R-CNN, with its ability to perform instance segmentation, is significantly better at distinguishing complex ship structures compared to Faster R-CNN, which primarily relies on bounding boxes. This is vital for identifying potential hazards.
  • Dataset Quality Matters: The creation of the MTMnet dataset, specifically from drone-captured footage, is a key contribution. It provides a more realistic and relevant benchmark for training AI models for maritime drone applications than existing datasets.
  • Environmental Factors Impact Performance: The paper acknowledges that adverse weather conditions, such as rain, can significantly degrade drone camera performance and, consequently, the AI's ability to recognize ships. This highlights the need for robust models and potentially data augmentation strategies.

Key Results

  • Mask R-CNN with R50-FPN 3x Backbone Achieves Best Segmentation: The study found that the Mask R-CNN model, when combined with the R50-FPN 3x backbone network and trained for 5000 iterations, achieved the highest performance in segmentation tasks.
  • High Accuracy Achieved: The Mask R-CNN model demonstrated an accuracy of approximately 95% after 5000 learning iterations, with a total loss value converging to around 0.3.
  • Faster R-CNN Limitations: While Faster R-CNN showed good performance in classifying simple ships, it struggled with complex structures, often misclassifying parts of a ship as separate entities.
  • Real-time Performance: The trained Mask R-CNN model was able to process video frames at approximately 10 frames per second, indicating potential for real-time application in drone navigation.

Limitations and Future Work

  • Small Ship Recognition: The current Mask R-CNN model sometimes fails to recognize very small ships, which is identified as an area for future research.
  • Adverse Weather Conditions: The performance degradation due to rain highlights the need for further research into data augmentation and models that are more resilient to environmental challenges.
  • Thin Structures and Wires: The paper notes that recognizing very fine structures like wires remains a challenge for AI.
  • Dataset Expansion: The authors suggest that collecting more diverse datasets and performing data augmentation are crucial for improving the robustness and automation of maritime delivery services using drones. They also advocate for continued research into recognizing more complex ship structures.

Summary updated

Abstract

The purpose of this paper is to study the recognition of ships and their structures to improve the safety of drone operations engaged in shore-to-ship drone delivery service. This study has developed a system that can distinguish between ships and their structures by using a convolutional neural network (CNN). First, the dataset of the Marine Traffic Management Net is described and CNN's object sensing based on the Detectron2 platform is discussed. There will also be a description of the experiment and performance. In addition, this study has been conducted based on actual drone delivery operations-the first air delivery service by drones in Korea.
BibTeX
@inproceedings{lim2022application,
  author = {Jae-Jun Lim and Dae‐Won Kim and Woon-Hee Hong and Min Gyu Kim and Dong-Hun Lee and Sun Young Kim and Jae Hoon Jeong},
  title = {{Application of Convolutional Neural Network (CNN) to Recognize Ship Structures}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22103824},
}
Figure