Real-Time Multi-Scene Visibility Enhancement for Promoting Navigational Safety of Vessels Under Complex Weather Conditions

Ryan Wen Liu, Yuxu Lu, Yuan Gao, Yu Guo, Wenqi Ren, Fenghua Zhu, Fei-Yue Wang

TL;DR

Researchers have developed ERANet, a lightweight neural network designed to clear up hazy, rainy, or low-light maritime images in real-time. By combining multi-directional edge detection with efficient structural reparameterization, the model achieves state-of-the-art restoration quality at over 60 FPS for 1080p video, significantly improving the reliability of automated vessel detection and navigation.

What Problem Does This Paper Address?

Navigating the open seas is dangerous enough without the added challenge of "blindness" caused by weather. While visible-light cameras are standard on modern vessels for Intelligent Waterborne Transportation Systems (IWTS), their performance plummets during haze, heavy rain, or at night.

The core challenge isn't just "fixing" the image; it's doing it universally and instantly. Most existing computer vision models are "specific-purpose"—a model trained to remove rain often fails miserably when faced with a low-light sunset. Furthermore, marine vessels often rely on edge computing devices with limited processing power. A high-accuracy model is useless if it can't process video frames fast enough to prevent a collision.

Methods

The authors introduce the Edge Reparameterization- and Attention-guided Neural Network (ERANet). Its architecture rests on three pillars:

  1. Kirsch-guided Reparameterization Module (KRM): Instead of standard edge detection (like Sobel), the authors use Kirsch operators that calculate gradients in eight different directions. This is crucial for identifying complex rain streaks and obscured vessel edges. To keep it fast, they use structural reparameterization: during training, the model uses multiple complex branches to learn these edges, but for inference (actual use), these branches are mathematically collapsed into a single, efficient convolutional layer.
  2. Dual Attention Mechanism: The network employs both a Channel Attention Module (CAM) to fix color distortions across RGB channels and a Spatial Attention Module (SAM) to focus the "brain power" of the network on critical regions, like distant ships on the horizon.
  3. Edge-guided Attention Residual Block (EARB): This is the basic building block of the network, integrating the KRM and attention modules into a lightweight residual structure.

The model is trained using a hybrid loss function that balances structural similarity (MS-SSIM), pixel-wise accuracy ($L_1$ loss), and spatial smoothness (Total Variation loss).

Datasets

The authors evaluated ERANet on a mix of general-purpose and maritime-specific datasets:

  • General Benchmarks: RESIDE-OTS (haze), Rain100L (rain), and LOL (low-light).
  • Maritime Specific: The Seaships dataset and the Singapore Maritime Dataset (SMD). Since "clear/degraded" image pairs are hard to find at sea, the authors synthesized weather effects onto these real-world maritime images to create a robust training ground.

Key Insights and Ideas

  • The Power of Eight Directions: Most vision models focus on horizontal and vertical gradients. By using Kirsch operators to look in eight directions, ERANet is significantly better at distinguishing between a vertical rain streak and the actual structural edge of a ship.
  • Training Complexity vs. Inference Speed: The use of reparameterization is the "aha moment" for deployment. It allows the model to benefit from the "wisdom" of a complex multi-branch architecture during training without paying the computational tax during real-time operation.
  • Downstream Synergy: The paper emphasizes that visibility enhancement isn't the end goal—it's a preprocessing step. They demonstrate that clearing the "visual noise" leads to immediate, massive gains in the accuracy of object detectors (like YOLOv7) and scene segmentors (like DeepLabv3+).

Key Results

  • Speed: ERANet is incredibly fast, taking only 0.016 seconds to process a 1080p image (roughly 62 FPS).
  • Efficiency: The model size is a mere 2.4 MB, making it ideal for the low-resource hardware found on autonomous surface vehicles.
  • Quality: In quantitative tests (PSNR and SSIM), ERANet consistently outperformed SOTA models like AirNet and TransWeather across all three weather conditions.
  • Practicality: When tested on the SMD dataset, ERANet improved the Mean Intersection over Union (mIoU) for scene segmentation from roughly 80-85% (with other methods) to 92.44%.

Limitations and Future Work

While ERANet is a powerhouse, the authors note it still struggles with extreme cases. If an image is "excessively dark" or contains local areas that are significantly brighter than the rest of the scene, the restoration can look unnatural.

Additionally, the Kirsch operators currently use a fixed scale. The authors suggest that future iterations should explore multi-scale features, as rain streaks and haze density can vary in size and "thickness" depending on the camera's focal length and the distance of the objects. Finally, they aim to move toward a Multi-Task Learning (MTL) framework where enhancement and object detection are handled simultaneously in a single, unified pipeline.

Summary updated

Abstract

The visible-light camera, which is capable of environment perception and navigation assistance, has emerged as an essential imaging sensor for marine surface vessels in intelligent waterborne transportation systems (IWTS). However, the visual imaging quality inevitably suffers from several kinds of degradations (e.g., limited visibility, low contrast, color distortion, etc.) under complex weather conditions (e.g., haze, rain, and low-lightness). The degraded visual information will accordingly result in inaccurate environment perception and delayed operations for navigational risk. To promote the navigational safety of vessels, many computational methods have been presented to perform visual quality enhancement under poor weather conditions. However, most of these methods are essentially specific-purpose implementation strategies, only available for one specific weather type. To overcome this limitation, we propose to develop a general-purpose multi-scene visibility enhancement method, i.e., edge reparameterization- and attention-guided neural network (ERANet), to adaptively restore the degraded images captured under different weather conditions. In particular, our ERANet simultaneously exploits the channel attention, spatial attention, and reparameterization technology to enhance the visual quality while maintaining low computational cost. Extensive experiments conducted on standard and IWTS-related datasets have demonstrated that our ERANet could outperform several representative visibility enhancement methods in terms of both imaging quality and computational efficiency. The superior performance of IWTS-related object detection and scene segmentation could also be steadily obtained after ERANet-based visibility enhancement under complex weather conditions.
BibTeX
@article{liu2024real,
  author = {Ryan Wen Liu and Yuxu Lu and Yuan Gao and Yu Guo and Wenqi Ren and Fenghua Zhu and Fei-Yue Wang},
  title = {{Real-Time Multi-Scene Visibility Enhancement for Promoting Navigational Safety of Vessels Under Complex Weather Conditions}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2024},
  doi = {10.1109/tits.2024.3454597},
}

Loading related content...

Figure