Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

Yu Guo, Ryan Wen Liu, Jingxiang Qu, Yuxu Lu, Fenghua Zhu, Yisheng Lv

TL;DR

This paper introduces DeepSORVF, a deep learning-based framework that fuses low-frequency AIS (Automatic Identification System) data with high-frequency video surveillance to track vessels in inland waterways. By shifting from simple point-in-time spatial matching to asynchronous trajectory matching, the system maintains accurate vessel identities even during complex maneuvers and severe occlusions.

What Problem Does This Paper Address?

Monitoring vessel traffic in busy inland waterways is a dual-sensor challenge.

  1. AIS Data provides "what" and "where" (vessel ID, speed, course) but is often asynchronous, suffers from time delays, and lacks visual context.
  2. Video Data provides rich visual "how" (appearance, real-time movement) but is "blind" to vessel identity and fails when ships pass behind one another (occlusion).

Existing fusion methods usually try to match a vessel's AIS position to a video detection at a single moment in time. This fails frequently because AIS and video frames rarely sync up perfectly, and GPS errors or "missing" AIS messages create "ghost" targets. The authors aim to bridge this gap by matching the entire path (trajectory) of the vessel rather than just its current coordinates.

Methods

The authors propose a framework called DeepSORVF (Deep learning-enabled asynchronous trajectory matching), which consists of three main pillars:

  • AIS Trajectory Extraction: Raw AIS data is cleaned, and missing points are predicted using forward geodetic computations. These coordinates are then projected from the world coordinate system (WCS) to the video's pixel coordinate system (PCS) using a pinhole camera model.
  • Anti-Occlusion Tracking: When a vessel is visually blocked by another ship or a bridge, the system uses "prior knowledge." It takes the last known velocity (from either AIS or previous video frames) to predict a "phantom" bounding box, allowing the tracker to maintain the vessel's ID until it reappears.
  • E-FastDTW (Enhanced Fast Dynamic Time Warping): This is the core innovation. Instead of measuring the Euclidean distance between two points, it calculates the similarity between two time-series trajectories. It considers the "warp path" (how the shapes align) and the angular direction of movement. The Hungarian Algorithm is then used to find the optimal global match between a set of AIS tracks and visual targets.

Datasets

The authors introduced a new benchmark dataset called FVessel.

  • Composition: 26 video sequences with corresponding synchronized AIS data.
  • Diversity: Captured at different locations (bridges, riversides) and under various weather/lighting conditions (sunny, cloudy, low-light).
  • Significance: Unlike generic MOT (Multi-Object Tracking) datasets, FVessel specifically targets the maritime domain where background movement (water) and vessel scale changes are unique challenges.

Key Insights and Ideas

  • Trajectories over Points: The "aha moment" here is that while a single GPS point might be noisy or laggy, the behavioral signature of a vessel (its path over 10–15 seconds) is unique and much harder to mismatch.
  • AIS as a "Visual" Aid: Usually, we think of video helping AIS. Here, the authors use AIS to help the video detector. By knowing where a ship should be according to its GPS, the system can "ignore" detection failures caused by overlapping ships.
  • Computational Efficiency: By using an association mechanism that only triggers a full trajectory similarity check when a "pairing" isn't already established, they kept the processing time low enough for real-time deployment.

Key Results

The DeepSORVF method was compared against several baselines, including Euclidean-based fusion (EDDF) and multi-source data fusion (MSDF).

  • Accuracy: DeepSORVF achieved an average Multi-Object Fusion Accuracy (MOFA) of 96.04%, significantly outperforming the 73.57% achieved by standard MSDF.
  • Robustness: In "Clip-04" (a particularly difficult scenario), the baseline methods dropped to ~54% accuracy, while DeepSORVF maintained 97.59%.
  • Speed: The system processes data at an average of 0.2562 seconds per frame, making it well-suited for real-time VTS (Vessel Traffic Service) applications.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • AIS Estimation Bias: The anti-occlusion bounding box prediction relies heavily on the accuracy of the AIS-reported speed and course. If the AIS data is significantly delayed or inaccurate, the predicted box may drift from the actual vessel position.
  • Perspective Challenges: When vessels move directly toward or away from the camera, their visual speed appears to change drastically due to perspective, which can sometimes confuse the trajectory matching.
  • Future Directions: The authors suggest incorporating more complex visual features (like vessel type or color) into the matching cost function to further harden the system against errors in extremely crowded ports.

Summary updated

Abstract

The automatic identification system (AIS) and video cameras have been widely exploited for vessel traffic surveillance in inland waterways. The AIS data could provide vessel identity and dynamic information on vessel position and movements. In contrast, the video data could describe the visual appearances of moving vessels without knowing the information on identity, position, movements, etc. To further improve vessel traffic surveillance, it becomes necessary to fuse the AIS and video data to simultaneously capture the visual features, identity, and dynamic information for the vessels of interest. However, the performance of AIS and video data fusion is susceptible to issues such as data spatial difference, message asynchronous transmission, visual object occlusion, etc. In this work, we propose a deep learning-based simple online and real-time vessel data fusion method (termed DeepSORVF). We first extract the AIS- and video-based vessel trajectories, and then propose an asynchronous trajectory matching method to fuse the AIS-based vessel information with the corresponding visual targets. In addition, by combining the AIS- and video-based movement features, we also present a prior knowledge-driven anti-occlusion method to yield accurate and robust vessel tracking results under occlusion conditions. To validate the efficacy of our DeepSORVF, we have also constructed a new benchmark dataset (termed FVessel) for vessel detection, tracking, and data fusion. It consists of many videos and the corresponding AIS data collected in various weather conditions and locations. The experimental results have demonstrated that our method is capable of guaranteeing high-reliable data fusion and anti-occlusion vessel tracking. The DeepSORVF code and FVessel dataset are publicly available at https://github.com/gy65896/DeepSORVF and https://github.com/gy65896/FVessel, respectively.
BibTeX
@article{guo2023asynchronous,
  author = {Yu Guo and Ryan Wen Liu and Jingxiang Qu and Yuxu Lu and Fenghua Zhu and Yisheng Lv},
  title = {{Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways}},
  journal = {IEEE Transactions on Intelligent Transportation Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=sUb8fCGtRH},
}

Loading related content...

Figure