Learning with Weak Annotations for Robust Maritime Obstacle Detection

Lojze Zust, Matej Kristan

Abstract

Robust maritime obstacle detection is crucial for safe navigation of autonomous boats and timely collision avoidance. The current state-of-the-art is based on deep segmentation networks trained on large datasets. Per-pixel ground truth labeling of such datasets, however, is labor-intensive and expensive. We propose a new scaffolding learning regime (SLR), that leverages weak annotations consisting of water edge, horizon and obstacle bounding boxes to train segmentation-based obstacle detection networks, and thus reduces the required ground truth labelling effort by twenty-fold. SLR trains an initial model from weak annotations, then alternates between re-estimating the segmentation pseudo labels and improving the network parameters. Experiments show that maritime obstacle segmentation networks trained using SLR on weak labels not only match, but outperform the same networks trained with dense ground truth labels, which is a remarkable result. In addition to increased accuracy, SLR also increases domain generalization and can be used for domain adaptation with a low manual annotation load.

Introduction

For autonomous surface vehicles (ASVs) to navigate safely, they must distinguish between navigable water and potential hazards with high precision. Modern solutions rely on deep semantic segmentation networks to classify every pixel in a camera feed as water, sky, or obstacle. While effective, these models are data-hungry. Creating the necessary training data is a significant bottleneck; manually drawing pixel-perfect masks for a single maritime image can take approximately 20 minutes.

Furthermore, standard segmentation training often treats all pixels with equal weight. For a maritime vessel, however, a false positive on the water's surface is far more critical than a misclassification high on a shoreline cliff. To address both the high cost of annotation and the need for navigation-specific accuracy, we introduce the Scaffolding Learning Regime (SLR).

The Scaffolding Learning Regime (SLR)

SLR is designed to train robust segmentation networks using only "weak" annotations: water-edge polylines, obstacle bounding boxes, and horizon lines (often estimated via IMU). This approach reduces manual labeling effort by a factor of 20. The regime follows a three-step process:

  1. Feature Warm-Up: The model is initially trained on partial labels. By applying domain knowledge—such as the physical constraint that water cannot exist above the horizon—the system creates masks where only high-confidence pixels are labeled. Auxiliary losses are employed to help the network learn the underlying shapes of objects within the provided bounding boxes.
  2. Pseudo-Label Estimation: Because large portions of the image remain unlabeled after the warm-up, the model uses its learned feature representations to "guess" the missing labels. It clusters similar features to generate dense pseudo-labels across the entire image.
  3. Fine-Tuning: The network undergoes a final round of training using these generated dense pseudo-labels, refining its ability to delineate complex maritime boundaries.

This "scaffolding" allows the network to internalize the structure of maritime scenes without requiring a human to trace every shoreline or wave-cut.

Experimental Setup and Results

We evaluated SLR using several key maritime datasets:

  • MaSTr1325: Used for training, containing 1,325 diverse maritime images.
  • MODS Benchmark: The primary evaluation set, focusing on a detection-oriented protocol within the "danger zone" (the area within 15 meters of the boat).
  • Singapore Marine Dataset (SMD): Used to test cross-domain generalization.

Performance Breakthroughs

The results were surprising: the SLR-trained model (WaSR-SLR) actually outperformed the version trained on dense, manually-annotated ground truth. On the MODS benchmark, WaSR-SLR improved the F1 score in the critical danger zone by 5.1 percentage points over the fully supervised baseline.

Robustness and Generalization

SLR demonstrated superior performance in two critical areas:

  • Reflection Handling: Maritime environments are plagued by sun glitter and boat reflections. SLR proved significantly more adept at ignoring these visual artifacts, which frequently cause "ghost" obstacles in standard models.
  • Cross-Domain Success: When tested on the Singapore Marine Dataset without additional tuning, the SLR model maintained higher accuracy than standard models. This suggests that training on weak labels prevents the model from overfitting to the specific textures of a single training environment, leading to better real-world utility.

Discussion and Limitations

The success of SLR suggests that weak annotations like bounding boxes may actually provide a better "teaching signal" for obstacle detection than dense masks. By focusing the model on object boundaries and the water-edge rather than pixel-matching, SLR aligns the learning process with the actual requirements of maritime navigation.

However, some challenges remain. The pseudo-labeling process can occasionally miss very thin structures, such as masts or the tips of surfboards. Additionally, highly transparent shallow water can still lead to false positives when the seabed is visible.

Conclusion

The Scaffolding Learning Regime represents a significant step forward in making autonomous maritime navigation more accessible and robust. By reducing the annotation burden twenty-fold while simultaneously improving detection performance and generalization, SLR proves that "weak" labels can indeed lead to stronger, more reliable vision systems. Future research will explore applying this scaffolding approach to other domains where dense labeling is prohibitively expensive, such as autonomous driving and medical imaging.

Summary updated

Abstract

Robust maritime obstacle detection is crucial for safe navigation of autonomous boats and timely collision avoidance. The current state-of-the-art is based on deep segmentation networks trained on large datasets. Per-pixel ground truth labeling of such datasets, however, is labor-intensive and expensive. We propose a new scaffolding learning regime (SLR), that leverages weak annotations consisting of water edge, horizon and obstacle bounding boxes to train segmentation-based obstacle detection networks, and thus reduces the required ground truth labelling effort by twenty-fold. SLR trains an initial model from weak annotations, then alternates between re-estimating the segmentation pseudo labels and improving the network parameters. Experiments show that maritime obstacle segmentation networks trained using SLR on weak labels not only match, but outperform the same networks trained with dense ground truth labels, which is a remarkable result. In addition to increased accuracy, SLR also increases domain generalization and can be used for domain adaptation with a low manual annotation load. The code and pre-trained models are available at https://github.com/lojzezust/SLR .
BibTeX
@inproceedings{zust2022learning,
  author = {Lojze Zust and Matej Kristan},
  title = {{Learning with Weak Annotations for Robust Maritime Obstacle Detection}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=U111kBx7Ef},
}
Figure