Efficient Water Segmentation with Transformer and Knowledge Distillation for USVs

Jingting Zhang, Jiantao Gao, Jinshuo Liang, Yiqiang Wu, Bin Li, Yang Zhai, Xiaomao LiJournal of Marine Science and Engineering 2023 · DOI

TL;DR

This paper tackles the challenge of accurately segmenting water for Unmanned Surface Vehicles (USVs) by proposing a novel method that fuses 2D camera images with 3D LiDAR data during training. Crucially, it then distills this multimodal knowledge into a lightweight 2D-only network for efficient real-time inference, achieving improved accuracy and significantly faster speeds compared to existing fusion methods.

What Problem Does This Paper Address?

Ensuring the safe navigation of Unmanned Surface Vehicles (USVs) heavily relies on accurately identifying the water surface. Traditional image-based methods struggle with reflections and lack 3D spatial understanding, leading to inaccuracies. While fusing 2D camera images with 3D LiDAR point clouds offers a more robust solution by combining rich visual information with precise depth data, this approach is computationally expensive and time-consuming, making it impractical for real-time applications on resource-constrained USVs. The core problem is bridging the gap between the accuracy of multimodal fusion and the efficiency required for real-time onboard processing.

Methods

The proposed method introduces two key innovations:

  1. Local and Non-Local Cross-Modality Fusion Module: This module leverages a transformer architecture to effectively fuse information from 2D images and 3D LiDAR point clouds. It first establishes point-pixel correspondence to align the modalities. Then, a transformer block with self-attention mechanisms is employed to capture both local and non-local relationships between features from both sensors, leading to a richer, more robust fused representation. This is a departure from simpler concatenation-based fusion methods that primarily consider local features.
  2. Multi-to-Single-Modality Knowledge Distillation (KD) Module: This module is designed for the training phase. It acts as a "teacher" by transferring the knowledge learned from the multimodal fusion process to a "student" 2D network. This distillation occurs at both the feature and prediction levels, ensuring that the 2D network learns to mimic the performance of the more complex multimodal system. The key advantage is that during inference, the 3D LiDAR processing branch can be entirely discarded, resulting in a significantly faster and more efficient 2D-only segmentation model.

The overall architecture involves a two-branch network (U-Net for 2D, PointNet++ for 3D) for initial feature extraction, followed by the proposed fusion module, and then the KD module to guide the 2D student network.

Datasets

The experiments were conducted on the CMWS [11] dataset, which consists of various scenes collected by USVs in water environments. This dataset is specifically designed for USV water segmentation tasks.

Key Insights and Ideas

  • Transformers for Cross-Modality Fusion: The paper demonstrates that transformer architectures, with their inherent ability to capture long-range dependencies via self-attention, are highly effective for fusing disparate sensor modalities like 2D images and 3D point clouds, outperforming simpler fusion strategies.
  • Knowledge Distillation for Efficiency: The core idea of distilling multimodal knowledge into a single-modal network is a powerful paradigm for achieving high performance without the computational burden of real-time multimodal processing. This allows for the deployment of advanced perception capabilities on resource-limited platforms.
  • The Value of 3D Data for Water Segmentation: Even though the final model is 2D-only, the study highlights that incorporating 3D LiDAR data during training significantly improves the accuracy of water segmentation, particularly in challenging near-shore scenarios where reflections and lighting conditions can be problematic for image-only methods.
  • Trade-off Between Training and Inference: The paper elegantly addresses the trade-off between training complexity and inference efficiency. By investing in a more complex training process with multimodal fusion and distillation, they achieve a highly efficient and accurate 2D inference model.

Key Results

  • Accuracy Improvement: The proposed method achieved approximately 1.5% improvement in accuracy (AC) and MaxF compared to classical image-based methods like U-Net, FCN, and DeepLabV3+.
  • Significant Speedup: The method achieved a dramatic speed improvement, increasing from 15 fps to 110 fps compared to the fusion method presented in [11], while maintaining comparable accuracy.
  • Qualitative Improvements: Qualitative results showed that the proposed method accurately segments water surfaces in various scenes, including challenging near-shore areas, outperforming the basic U-Net baseline which struggled with water reflection and adverse weather.
  • Ablation Study: The ablation study confirmed the effectiveness of both the cross-modality fusion module and the knowledge distillation module, with the combined approach yielding the best results (95.91% AC and 92.10% mIoU).

Limitations and Future Work

  • Dataset Size and Generalization: The authors acknowledge that the CMWS dataset, while useful, might be limited in size and collected in inland waterways. This could potentially lead to overfitting and affect the model's generalization ability to diverse sea surface environments.
  • Future Work Directions:
    • Semisupervised Methods: Exploring semisupervised learning to address dataset scarcity and further optimize USV water segmentation.
  • Additional Data Sources: Investigating the integration of other sensors like radar or sonar to enhance accuracy and efficiency.
  • Transfer Learning: Leveraging transfer learning techniques to improve performance across different water scenes and environments.
  • Reinforcement Learning: Applying reinforcement learning to optimize the model in dynamic and changing water conditions.
  • Real-time System Integration: Developing a real-time water segmentation system for practical USV applications in scenarios like search and rescue or environmental monitoring.

Summary updated

Abstract

Water segmentation is a critical task for ensuring the safety of unmanned surface vehicles (USVs). Most existing image-based water segmentation methods may be inaccurate due to light reflection on the water. The fusion-based method combines the paired 2D camera images and 3D LiDAR point clouds as inputs, resulting in a high computational load and considerable time consumption, with limits in terms of practical applications. Thus, in this study, we propose a multimodal fusion water segmentation method that uses a transformer and knowledge distillation to leverage 3D LiDAR point clouds in order to assist in the generation of 2D images. A local and non-local cross-modality fusion module based on a transformer is first used to fuse 2D images and 3D point cloud information during the training phase. A multi-to-single-modality knowledge distillation module is then applied to distill the fused information into a pure 2D network for water segmentation. Extensive experiments were conducted with a dataset containing various scenes collected by USVs in the water. The results demonstrate that the proposed method achieves approximately 1.5% improvement both in accuracy and MaxF over classical image-based methods, and it is much faster than the fusion-based method, achieving speeds ranging from 15 fps to 110 fps.
BibTeX
@article{zhang2023efficient,
  author = {Jingting Zhang and Jiantao Gao and Jinshuo Liang and Yiqiang Wu and Bin Li and Yang Zhai and Xiaomao Li},
  title = {{Efficient Water Segmentation with Transformer and Knowledge Distillation for USVs}},
  journal = {Journal of Marine Science and Engineering},
  year = {2023},
  doi = {10.3390/jmse11050901},
}
Figure