Abstract
Effective vessel traffic surveillance in inland waterways requires the integration of diverse data sources. While the Automatic Identification System (AIS) provides critical identity and dynamic information, video surveillance offers rich visual context. However, fusing these sources is often hampered by asynchronous messaging, data gaps, and environmental occlusions. This paper presents DeepSORVF, a deep learning-enabled framework that utilizes asynchronous trajectory matching to fuse AIS and video data. By shifting from point-to-point spatial matching to a trajectory-based approach using Enhanced Fast Dynamic Time Warping (E-FastDTW), the system achieves high-reliability tracking even under complex conditions. To validate this approach, the authors introduced the FVessel benchmark dataset, demonstrating that DeepSORVF significantly outperforms traditional fusion methods in both accuracy and robustness.
Introduction
Monitoring vessel traffic in busy inland waterways is essential for safety and logistics. Currently, surveillance relies on two primary sensors: AIS and video cameras. AIS data provides the "what" and "where"—vessel identity, speed, and course—but is often low-frequency and lacks visual confirmation. Conversely, video data provides the "how"—real-time visual appearance and movement—but cannot inherently identify vessels or maintain tracking when ships are occluded by bridges or other vessels.
Traditional fusion methods typically attempt to match a vessel's AIS position to a video detection at a single timestamp. This approach frequently fails because AIS and video frames are rarely synchronized, and GPS errors often create "ghost" targets. DeepSORVF addresses these limitations by matching the entire behavioral signature (trajectory) of a vessel rather than isolated coordinates.
Methodology
The DeepSORVF framework consists of three integrated components designed to handle the complexities of maritime environments.
1. AIS Trajectory Extraction and Projection
Raw AIS data is often noisy or incomplete. The system first cleans the data and predicts missing points using forward geodetic computations. To align this data with video frames, coordinates are projected from the World Coordinate System (WCS) to the video's Pixel Coordinate System (PCS) using a pinhole camera model. This transformation allows the system to compare GPS-derived paths directly with visual movement on the screen.
2. E-FastDTW for Trajectory Matching
The core innovation of DeepSORVF is the use of Enhanced Fast Dynamic Time Warping (E-FastDTW). Unlike standard Euclidean distance measurements, which require perfectly synchronized points, E-FastDTW calculates the similarity between two time-series trajectories by finding an optimal "warp path." This method accounts for variations in sampling rates and temporal shifts. By incorporating angular direction into the similarity cost and using the Hungarian Algorithm for global optimization, the system ensures that each visual target is correctly paired with its corresponding AIS identity.
3. Prior Knowledge-Driven Anti-Occlusion
Occlusion is a major hurdle in maritime tracking. When a vessel is visually blocked, the system leverages "prior knowledge"—specifically the last known velocity from either AIS or previous video frames. It generates a "phantom" bounding box to maintain the vessel's track. This allows the system to preserve identity continuity until the vessel reappears, preventing the tracker from assigning a new ID or losing the target entirely.
The FVessel Dataset
To evaluate the framework, the authors developed FVessel, a specialized benchmark dataset for maritime data fusion. It includes:
- 26 video sequences with synchronized AIS data.
- Diverse environments, including various weather conditions (sunny, cloudy, low-light) and camera locations (bridges, riversides).
- Domain-specific challenges, such as significant scale changes as vessels approach the camera and background noise from moving water.
Experimental Results
DeepSORVF was benchmarked against standard methods, including Euclidean-based fusion (EDDF) and multi-source data fusion (MSDF).
- Superior Accuracy: DeepSORVF achieved an average Multi-Object Fusion Accuracy (MOFA) of 96.04%, a substantial improvement over the 73.57% achieved by MSDF.
- Robustness in Complex Scenarios: In highly challenging sequences (e.g., "Clip-04"), where baseline methods dropped to approximately 54% accuracy due to heavy traffic and occlusions, DeepSORVF maintained a high performance of 97.59%.
- Real-Time Performance: The system processes data at an average rate of 0.2562 seconds per frame, meeting the requirements for real-time Vessel Traffic Service (VTS) applications.
Discussion and Future Work
While DeepSORVF represents a significant advancement, certain challenges remain. The accuracy of the anti-occlusion bounding box is dependent on the quality of AIS-reported speed and course; significant AIS lag can lead to "drift" in the predicted position. Additionally, perspective distortion—where vessels moving directly toward the camera appear to change speed rapidly in pixel space—can occasionally complicate trajectory matching.
Future research will explore integrating additional visual features, such as vessel type, color, and size, into the matching cost function. This multi-feature approach aims to further harden the system against errors in extremely crowded port environments where trajectory signatures may overlap.
Conclusion
By moving from instantaneous spatial matching to asynchronous trajectory alignment, DeepSORVF provides a robust solution for multimodal maritime surveillance. The combination of E-FastDTW and prior knowledge-driven tracking allows for high-accuracy vessel identification and monitoring, even in the presence of sensor asynchronicity and physical occlusions.
Summary updated