Fast Image-Based Obstacle Detection From Unmanned Surface Vehicles

Matej Kristan, Vildana Sulic Kenk, Stanislav Kovacic, Janez Pers

This paper presents a highly efficient approach to real-time obstacle detection for small Unmanned Surface Vehicles (USVs) operating in complex coastal environments. By combining a novel graphical model with weak semantic priors, the authors achieve state-of-the-art performance on a single CPU core without needing specialized hardware or computationally expensive feature extraction.

TL;DR

The authors propose a fast, segmentation-based obstacle detection algorithm for USVs that models the marine environment as three vertically stacked semantic regions (sky, land/haze, and water). By using a Markov Random Field (MRF) and an extremely efficient EM-like optimization, the system runs at over 70fps on a single core while outperforming more complex methods on a newly introduced large-scale marine dataset.

What Problem Does This Paper Address?

Navigating small USVs in coastal waters is surprisingly difficult for computer vision. Unlike ground vehicles, which can often assume a stable "ground plane," USVs contend with a constantly shifting horizon, dynamic obstacles (buoys, swimmers, other boats), and challenging lighting conditions like sun glitter on the water.

Furthermore, small USVs have strict constraints on weight and power, meaning they can't always carry heavy GPUs or active sensors like LIDAR. Existing vision-based methods often struggle with real-time performance or rely on brittle assumptions, such as a perfectly straight horizon line, which fails in marinas or near complex shorelines.

Methods

The core of the approach is a semantic generative model that treats the image as a mixture of four components: three Gaussian Mixture Models (GMMs) representing the dominant regions (sky, middle/land, and water) and a uniform distribution to capture "outliers"—which are the obstacles.

Key technical highlights include:

  • Weak Structural Priors: Instead of complex texture features, the model uses simple color and pixel coordinates. It imposes "weak priors" (essentially educated guesses) that the sky is generally at the top, land in the middle, and water at the bottom.
  • Markov Random Field (MRF): To ensure the segmentation is spatially smooth (i.e., neighboring pixels are likely to belong to the same class), the authors place an MRF over the priors and posteriors.
  • Efficient Optimization: The authors derive a specialized Expectation-Maximization (EM) algorithm. Crucially, they show that the neighborhood interactions in the MRF can be computed using simple convolutions, allowing the entire optimization to run incredibly fast.
  • Multi-scale Processing: The algorithm performs its main segmentation on a drastically reduced 50x50 pixel image and then rescales the results, which provides a massive speed boost without sacrificing the detection of dangerous (larger/closer) obstacles.
The proposed pipeline: from input image to obstacle map.

Datasets

The authors introduced the Marine Obstacle Detection Dataset (MODD), which at the time of publication was the largest annotated dataset of its kind.

  • Scale: 12 video sequences with 4,454 fully annotated frames (640x480 resolution).
  • Diversity: Captured in the Gulf of Trieste, Slovenia, across different times of day and weather conditions.
  • Challenges: Includes "normal" conditions and "extreme" conditions, specifically sequences where the USV faces the sun, creating intense glitter and contrast changes.

Key Insights and Ideas

  • Semantic Structure over Texture: You don't need deep, computationally expensive texture features for marine obstacle detection. The inherent vertical structure of the marine environment (sky-land-water) provides enough constraint if modeled correctly.
  • Outliers are the Goal: By explicitly modeling obstacles as "outliers" that don't fit the three main semantic GMMs, the system can detect a wide variety of objects (buoys, divers, debris) without needing a specific detector for every possible object type.
  • Convolutions for Speed: The "aha moment" for performance is realizing that MRF-based spatial smoothing can be implemented via image convolutions during the EM process, bypassing the need for slower graph-cut optimizations.
The graphical model used for semantic segmentation.

Key Results

  • Speed: The Matlab implementation runs at over 70fps on a single-core 3.06 GHz Intel Xeon. The segmentation itself takes only about 10ms per frame.
  • Accuracy: In "normal" conditions, the proposed Semantic Segmentation Model (SSM) achieved an F-score of 0.819, significantly outperforming alternatives like Graph-Cut Models (GCM) and standard MRF approaches.
  • Robustness: Even in "extreme" sun glitter conditions, the model remained robust, maintaining a low false-positive rate compared to other methods that were overwhelmed by the reflections.
Qualitative comparison showing the proposed method (SSM) producing much cleaner segmentation than competing approaches.

Limitations and Future Work

While highly effective, the system has some acknowledged weaknesses:

  • Extreme Glitter: Very low-lying sun directly in front of the camera can still cause enough saturated glitter to be misidentified as an obstacle.
  • Reflections and Blurs: Strong reflections of landmasses in very still water or extremely blurred transitions between sea and sky can lead to conservative (upward-shifted) water-edge estimation.
  • Future Directions: The authors suggest integrating Inertial Measurement Unit (IMU) data to provide a better initial estimate of the horizon and porting the algorithm to GPUs to exploit its convolutional nature for even higher speeds.

Summary updated

Abstract

Obstacle detection plays an important role in unmanned surface vehicles (USVs). The USVs operate in a highly diverse environments in which an obstacle may be a floating piece of wood, a scuba diver, a pier, or a part of a shoreline, which presents a significant challenge to continuous detection from images taken on board. This paper addresses the problem of online detection by constrained, unsupervised segmentation. To this end, a new graphical model is proposed that affords a fast and continuous obstacle image-map estimation from a single video stream captured on board a USV. The model accounts for the semantic structure of marine environment as observed from USV by imposing weak structural constraints. A Markov random field framework is adopted and a highly efficient algorithm for simultaneous optimization of model parameters and segmentation mask estimation is derived. Our approach does not require computationally intensive extraction of texture features and comfortably runs in real time. The algorithm is tested on a new, challenging, dataset for segmentation, and obstacle detection in marine environments, which is the largest annotated dataset of its kind. Results on this dataset show that our model outperforms the related approaches, while requiring a fraction of computational effort.
BibTeX
@article{kristan2016fast,
  author = {Matej Kristan and Vildana Sulic Kenk and Stanislav Kovacic and Janez Pers},
  title = {{Fast Image-Based Obstacle Detection From Unmanned Surface Vehicles}},
  journal = {IEEE Trans. Cybernetics},
  year = {2016},
  doi = {10.1109/tcyb.2015.2412251},
}

Loading related content...

Figure