Learning with Weak Annotations for Robust Maritime Obstacle Detection

Lojze Zust, Matej Kristan

TL;DR

Researchers have developed a "Scaffolding Learning Regime" (SLR) that trains maritime obstacle detection models using weak annotations—like bounding boxes and horizon lines—instead of expensive per-pixel masks. Surprisingly, this approach reduces manual labeling effort by 20x while actually outperforming models trained on dense, fully supervised data.

What Problem Does This Paper Address?

For autonomous boats to navigate safely, they need to know exactly where the water ends and obstacles begin. Currently, the best way to do this is through semantic segmentation (labeling every pixel as water, sky, or obstacle).

However, there is a massive bottleneck: data annotation. Manually drawing pixel-perfect masks for a single maritime image takes about 20 minutes. Furthermore, standard segmentation training treats all pixels equally, but for a boat, a mistake on the water's surface (a false positive) is much more disruptive than a mistake high up on a shoreline cliff. The authors aim to bridge the gap between the need for massive datasets and the high cost of high-quality labels.

Methods

The authors introduce the Scaffolding Learning Regime (SLR), a three-step process that replaces dense masks with "weak" annotations (water-edge polylines, obstacle bounding boxes, and IMU-estimated horizon lines).

  1. Feature Warm-Up: The model is first trained on "partial labels." Using domain knowledge (e.g., "water cannot be above the horizon"), the researchers create masks where only certain pixels are labeled with high confidence. They also use auxiliary losses to force the network to learn the shapes of objects inside bounding boxes.
  2. Pseudo-Label Estimation: Since many pixels are still unlabeled, the model uses its learned features to "guess" the missing labels. It clusters similar features together to create dense pseudo-labels for the entire image.
  3. Fine-Tuning: Finally, the model is fine-tuned using these dense pseudo-labels.

This "scaffolding" allows the network to learn the complex structure of maritime scenes without a human ever having to trace a single shoreline.

Datasets

  • MaSTr1325: Used for training; contains 1,325 diverse maritime images. The authors added weak annotations (water edges and boxes) to this set.
  • MODS Benchmark: The primary evaluation set, which uses a detection-oriented protocol focusing on the "danger zone" (within 15 meters of the boat).
  • Singapore Marine Dataset (SMD): Used to test "cross-domain generalization"—essentially seeing if a model trained in one part of the world works in another without extra tuning.

Key Insights and Ideas

  • Less is More: The most striking finding is that SLR-trained models outperformed fully supervised ones. Why? Because weak annotations (like bounding boxes) force the model to focus on object detection rather than just matching pixel textures.
  • Navigation-Centric Learning: Standard segmentation often wastes "learning capacity" on irrelevant boundaries (like where the shore meets the sky). SLR focuses the model’s attention on the water edge and dynamic obstacles—the things that actually matter for not crashing.
  • Robustness to Reflections: Maritime environments are full of "visual noise" like sun glitter and boat reflections. SLR proved significantly better at ignoring these reflections, which often trip up standard models and cause "ghost" obstacles.

Key Results

  • Superior Performance: On the MODS benchmark, WaSR-SLR (the authors' model) improved the F1 score in the critical "danger zone" by 5.1 percentage points over the fully supervised version.
  • Efficiency: The labeling effort was reduced by a factor of 20, yet the model achieved a new state-of-the-art result.
  • Generalization: When tested on the Singapore Marine Dataset, the SLR model maintained much higher accuracy than the standard model, proving it is less likely to "overfit" to the specific look of the training waters.

Limitations and Future Work

While SLR is robust, it isn't perfect. The authors noted that the model sometimes struggles with very thin structures, such as masts or the ends of surfboards, which might be missed by the pseudo-labeling process. Additionally, objects visible through shallow, clear water can still trigger false positives.

Future work involves applying this scaffolding approach to other domains, such as autonomous driving or medical imaging, where per-pixel labeling is similarly expensive and domain constraints (like "the road is usually below the sky") can be mathematically defined.

Summary updated

Abstract

Robust maritime obstacle detection is critical for safe navigation of autonomous boats and timely collision avoidance. The current state-of-the-art is based on deep segmentation networks trained on large datasets. However, per-pixel ground truth labeling of such datasets is labor-intensive and expensive. We propose a new scaffolding learning regime (SLR) that leverages weak annotations consisting of water edges, the horizon location, and obstacle bounding boxes to train segmentation-based obstacle detection networks, thereby reducing the required ground truth labeling effort by a factor of twenty. SLR trains an initial model from weak annotations and then alternates between re-estimating the segmentation pseudo-labels and improving the network parameters. Experiments show that maritime obstacle segmentation networks trained using SLR on weak annotations not only match but outperform the same networks trained with dense ground truth labels, which is a remarkable result. In addition to the increased accuracy, SLR also increases domain generalization and can be used for domain adaptation with a low manual annotation load. The SLR code and pre-trained models are freely available online.
BibTeX
@inproceedings{zust2022learning,
  author = {Lojze Zust and Matej Kristan},
  title = {{Learning with Weak Annotations for Robust Maritime Obstacle Detection}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=U111kBx7Ef},
}

Loading related content...

Figure