Graph Learning-Driven Multi-Vessel Association: Fusing Multimodal Data for Maritime Intelligence

Yuxu Lu, Kaisen Yang, Dong Yang, Haifeng Ding, Jinxian Weng, Ryan Wen LiuarXiv 2025 · DOI · arXiv

Abstract

Effective maritime surveillance requires the integration of heterogeneous data sources, primarily the Automatic Identification System (AIS) and Closed-Circuit Television (CCTV). However, associating digital AIS identities with visual targets in video feeds is challenging due to coordinate disparities, sampling asynchronicity, and environmental noise. This paper presents GMvA, a graph learning-driven multi-vessel association framework. GMvA models vessel interactions as dynamic graphs and utilizes spatial-temporal attention mechanisms to capture complex motion patterns. Furthermore, an uncertainty-aware fusion module is introduced to handle measurement noise and occlusions. Evaluated on real-world datasets from the Huangpu River, GMvA achieves an association accuracy of 92.28%, demonstrating superior robustness in crowded waterways and scenarios with missing data.

Introduction

Modern maritime management relies on a combination of digital tracking and visual monitoring. The Automatic Identification System (AIS) provides precise GPS-based coordinates and vessel identification, while CCTV systems offer high-fidelity visual evidence for safety and security. Fusing these two modalities—matching a digital AIS signature to a specific physical vessel on screen—is a critical task for maritime intelligence.

Overview of the maritime monitoring environment and the multi-stage association pipeline.

Despite its importance, multi-vessel association faces several hurdles. AIS and CCTV data differ in dimensionality and sampling rates. Environmental factors, such as vessel occlusions in crowded ports and unstable AIS signals, further complicate the matching process. Traditional distance-based methods often fail when vessels are in close proximity or when projection errors occur during coordinate transformation.

Methodology: The GMvA Framework

The proposed Graph learning-driven Multi-vessel Association (GMvA) method addresses these challenges through a multi-stage pipeline designed to learn robust spatiotemporal features and account for data uncertainty.

The GMvA framework architecture, detailing data preprocessing, graph-driven feature learning, and multi-vessel association.

1. Coordinate Unification and Preprocessing

The first step involves mapping CCTV pixel coordinates into the real-world AIS coordinate system. A Linear Regression Model (LRM) is employed to establish a unified reference frame. Simultaneously, AIS data is cleaned and interpolated to ensure temporal alignment with the video frames.

2. Spatiotemporal Graph Learning

Rather than treating vessels as independent entities, GMvA constructs dynamic graphs where nodes represent vessel states and edges capture temporal and spatial relationships.

  • Temporal Graph Attention (TGA) Layer: This layer models the evolution of individual vessel trajectories over time, capturing motion dynamics.
  • Spatial-Temporal Attention (STA) Block: This module uses multi-head attention to analyze interactions between multiple vessels. By considering the "social context" of the waterway—how vessels move relative to one another—the system can better distinguish between similar trajectories in high-density areas.

3. Uncertainty-Aware Fusion (MLP-UMF)

A key innovation of GMvA is the MLP-based Uncertainty Fusion Module. Instead of producing a deterministic similarity score, the module models the similarity between AIS and video targets as a Gaussian distribution. By predicting both a mean similarity and a variance (uncertainty), the system can effectively penalize low-confidence matches caused by noise or occlusions, enhancing overall robustness.

4. Global Association

The final association is performed using the Hungarian algorithm, which optimizes the global matching matrix to ensure a consistent one-to-one correspondence between the two datasets.

Experimental Evaluation

The researchers evaluated GMvA using real-world data collected from the Huangpu River in Shanghai, China. The dataset includes various traffic densities (Low, Moderate, and High) and manually annotated ground truth for validation.

Key Results

GMvA demonstrated significant performance gains over baseline models, including LSTMs and Transformers:

  • Accuracy: GMvA achieved an average accuracy of 92.28%, outperforming the Transformer baseline (82.90%).
  • Density Robustness: In high-density scenarios with approximately 30 vessels, GMvA maintained an accuracy of 86.73%, whereas traditional methods like Dynamic Time Warping (DTW) dropped to 27.64%.
  • Resilience to Data Loss: The graph-based approach allows the model to maintain 45.45% accuracy even when 10 consecutive AIS records are missing, a scenario where most other models fail completely.

Conclusion

GMvA provides a robust solution for maritime multi-vessel association by leveraging graph learning and uncertainty modeling. By treating the waterway as a dynamic system of interacting entities rather than isolated points, the framework achieves high accuracy even in challenging, real-world conditions. Future work will focus on integrating radar data for further redundancy and developing online learning capabilities for real-time adaptation to new environments.

Summary updated

Abstract

Ensuring maritime safety and optimizing traffic management in increasingly crowded and complex waterways require effective waterway monitoring. However, current methods struggle with challenges arising from multimodal data, such as dimensional disparities, mismatched target counts, vessel scale variations, occlusions, and asynchronous data streams from systems like the automatic identification system (AIS) and closed-circuit television (CCTV). Traditional multi-target association methods often struggle with these complexities, particularly in densely trafficked waterways. To overcome these issues, we propose a graph learning-driven multi-vessel association (GMvA) method tailored for maritime multimodal data fusion. By integrating AIS and CCTV data, GMvA leverages time series learning and graph neural networks to capture the spatiotemporal features of vessel trajectories effectively. To enhance feature representation, the proposed method incorporates temporal graph attention and spatiotemporal attention, effectively capturing both local and global vessel interactions. Furthermore, a multi-layer perceptron-based uncertainty fusion module computes robust similarity scores, and the Hungarian algorithm is adopted to ensure globally consistent and accurate target matching. Extensive experiments on real-world maritime datasets confirm that GMvA delivers superior accuracy and robustness in multi-target association, outperforming existing methods even in challenging scenarios with high vessel density and incomplete or unevenly distributed AIS and CCTV data.
BibTeX
@misc{lu2025graph,
  author = {Yuxu Lu and Kaisen Yang and Dong Yang and Haifeng Ding and Jinxian Weng and Ryan Wen Liu},
  title = {{Graph Learning-Driven Multi-Vessel Association: Fusing Multimodal Data for Maritime Intelligence}},
  year = {2025},
  eprint = {2504.09197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.09197},
}
Figure