Abstract
Autonomous ships are expected to improve the level of safety and efficiency in future maritime navigation. Such vessels need perception for two purposes: to perform autonomous situational awareness and to monitor the integrity of the sensor system itself. In order to meet these needs, the perception system must fuse data from novel and traditional perception sensors using Artificial Intelligence (AI) techniques. This article overviews the recognized operational requirements that are imposed on regular and autonomous seafaring vessels, and then proceeds to consider suitable sensors and relevant AI techniques for an operational sensor system. The integration of four sensors families is considered: sensors for precise absolute positioning, visual sensors, audio sensors, and sensors for remote-sensing. Additionally, sources of auxiliary data, such as Automatic Identification System (AIS) and external data archives are discussed. The perception tasks are related to well-defined problems, such as situational abnormality detection, vessel classification, and localization, that are solvable using AI techniques. Machine learning methods, such as deep learning and Gaussian processes, are identified to be especially relevant for these problems.
Introduction
While autonomous vehicles on land dominate the headlines, the maritime industry faces a unique set of challenges. Maritime Autonomous Surface Ships (MASS) must operate in incredibly harsh environments—including polar ice, heavy fog, and salt spray—where traditional sensors often struggle. Currently, human navigators rely on a mix of RADAR, GNSS, and visual observation, but they often distrust individual electronic devices due to potential inaccuracies. To achieve true autonomy, a ship needs a perception system that doesn't just "see" objects, but also cross-references multiple data sources to ensure system integrity. The challenge lies in defining the operational requirements for these systems and identifying which AI techniques can handle the massive, multi-modal data streams generated at sea.
System Architecture and Sensor Fusion
The proposed architecture for an autonomous perception system is modular, focusing on the fusion of four key sensor families to provide a comprehensive view of the vessel's surroundings. This assembly integrates deployed sensors with auxiliary data sources, feeding into AI and machine learning algorithms to produce situational awareness and monitor system integrity.
The sensor families include:
- Precise Positioning: Combining GNSS with Inertial Measurement Units (IMU) to maintain centimeter-level accuracy even during dynamic ship maneuvers.
- Visual Sensors: Using monocular and stereo cameras for object classification and distance estimation.
- Remote Sensing: Leveraging the long-range reliability of RADAR with the high-resolution, close-range detail of LiDAR.
- Audio Sensors: Utilizing microphone arrays to detect "blind spot" vessels or those not broadcasting their position via AIS (Automatic Identification System).
Detection Regimes and Sensor Ranges
A critical insight from the research is the "regime" split strategy for sensor fusion. Long-range detection (typically over 1 nautical mile) relies heavily on RADAR and AIS messages. In contrast, close-range navigation (under 1 nautical mile) prioritizes high-resolution data from LiDAR, stereo cameras, and audio sensors. Audio sensors are particularly noted as an "underdog" modality, critical for detecting small boats or situational abnormalities (like a person falling overboard) that cameras might miss in poor visibility.
AI Techniques and Datasets
On the software side, Deep Learning (DL)—specifically Convolutional Neural Networks (CNNs) like SSD and Faster R-CNN—is identified as the gold standard for object detection. For trajectory analysis and handling the inherent uncertainty of maritime environments, Gaussian Processes (GPs) provide a robust probabilistic framework.
To train these models, the review highlights several public maritime datasets, such as SeaShips and MARVEL for classification, VAIS for paired visible/infrared imagery, and ShipsEar for acoustic research. The authors also validated vision-based recognition using their own experimental results from the Helsinki–Tallinn ferry line, demonstrating that AI can successfully recognize ships even in low-brightness and ill-exposed conditions.
Operational Requirements and Key Results
The paper consolidates target values for autonomous ship performance, setting a benchmark for future development:
- Positioning Accuracy: A target horizontal accuracy of <3 meters for port and coastal operations.
- Detection Range: Requirements vary by object size, such as detecting large ships at 8 NM and small kayak-sized objects at 1 NM.
- AI Latency: Situational awareness results should be computed in "online" mode, ideally within 60 seconds.
Challenges and Future Outlook
The primary limitation remains the maritime environment itself. Rain, fog, and snow significantly degrade the performance of LiDAR and cameras. Furthermore, the high computational cost of processing high-dimensional video and audio data in real-time remains a hurdle for shipboard hardware. Future research is directed toward improved sensor fusion algorithms that can better handle conflicting data, wind noise mitigation for audio sensors, and the development of a unified regulatory framework for autonomous maritime safety.
Summary updated