Robust wind turbine blade segmentation from RGB images in the wild

Raül Pérez-Gonzalo, Andreas Espersen, Antonio Agudo

TL;DR

This paper introduces a robust algorithm for segmenting wind turbine blades from diverse, real-world drone imagery. By combining a customized U-Net with novel post-processing steps like hole-filling and an on-the-fly random forest, they achieve state-of-the-art accuracy, crucial for automated wind turbine maintenance.

What Problem Does This Paper Address?

The wind energy industry is rapidly expanding, necessitating efficient and automated methods for wind turbine maintenance. A critical first step in this process is accurately identifying the turbine blades in images captured during inspections. Current methods often struggle with the variability of real-world conditions – different lighting, blade shapes, sizes, and the presence of shadows or dirt – leading to unreliable segmentation. This unreliability hinders the development of automated systems for tasks like defect detection, which are vital for cost-effective and safe operation. This paper aims to bridge this gap by providing a robust segmentation algorithm that generalizes well to diverse "in-the-wild" images.

Methods

The proposed approach builds upon a modified U-Net architecture, termed Blade U-Net (BU-Net). This network is trained with a custom loss function that combines a focal loss (to penalize misclassifications and focus on harder examples) with a contiguity regularization term (to encourage smoother, more connected segmentations).

However, the core novelty lies in the post-processing pipeline designed to refine the BU-Net's output:

  1. Hole Filling (First Pass): This step addresses internal holes within the segmented blade regions and also fills gaps along the image borders. It leverages detected blade orientation to enforce continuity.
  2. Unsupervised Random Forest: An on-the-fly random forest is trained for each blade surface. This module acts as a regularizer, learning the typical color intensities and local neighborhoods associated with the blade, thereby correcting misplaced pixels and improving fine-grained boundary precision.
  3. Hole Filling (Second Pass): A final hole-filling step is applied to refine the output from the random forest, further enhancing the segmentation's completeness.

This multi-stage approach aims to leverage both learned features from the U-Net and explicit geometric and contextual priors.

Datasets

The authors introduce a new dataset consisting of 1,712 images for training, 120 for validation, and 200 for testing. These images are high-resolution and captured by drones and ground-based equipment from various locations, exhibiting a wide range of blade appearances and environmental conditions. The dataset was manually annotated, and the authors emphasize that the training, validation, and test sets are composed of images from different windfarms and inspection campaigns to ensure the model's generalizability.

Key Insights and Ideas

  • Post-processing is Crucial: While a well-tuned U-Net can achieve decent results, the paper strongly demonstrates that post-processing steps are essential for achieving robust and accurate segmentation in challenging real-world scenarios. Each step in their pipeline contributes significantly to the final performance.
  • Leveraging Domain Knowledge: The algorithm effectively incorporates prior knowledge about wind turbine blades, such as their tendency to traverse images and the continuity of their edges. This is evident in the hole-filling steps and the random forest's ability to learn blade-specific patterns.
  • On-the-Fly Random Forest for Adaptability: Training a random forest "on-the-fly" for each blade surface allows the model to adapt to specific image features and variations within that blade, leading to more personalized and accurate corrections.
  • Generality Across Windfarms: The proposed method shows strong performance across images from different windfarms, highlighting its robustness and ability to generalize beyond the specific training data's origin.

Key Results

The proposed algorithm achieves a remarkable 97.39% accuracy, 93.35% recall, and 95.73% F1-score on their challenging test dataset. This represents a significant improvement over the baseline BU-Net (92.28% accuracy, 85.73% recall) and outperforms other state-of-the-art methods like U-Net, DeepLabv3+, and ResNeSt. The paper also shows that each post-processing step contributes meaningfully to the overall performance, with the first hole-filling step providing the largest gain in recall. Qualitative results in Figure 4 visually confirm the progressive improvement in segmentation quality through each stage of the pipeline.

Limitations and Future Work

The authors acknowledge that their dataset includes a few instances of the wind turbine hub, and the BU-Net occasionally struggles with these. Additionally, distinct sources of contamination or broad color intensity variations within the blade region can sometimes mislead the segmentation.

Future work includes extending the model to semantic segmentation, where different parts of the wind turbine (beyond just the blade) need to be identified. They also aim to study the impact of their robust segmentation on downstream tasks, such as defect detection. The paper also notes that segmenting the tower as blade is desirable, but distinguishing turbines in the landscape from the actual blade is a challenging problem for future investigation.

Summary updated

Abstract

With the relentless growth of the wind industry, there is an imperious need to design automatic data-driven solutions for wind turbine maintenance. As structural health monitoring mainly relies on visual inspections, the first stage in any automatic solution is to identify the blade region on the image. Thus, we propose a novel segmentation algorithm that strengthens the U-Net results by a tailored loss, which pools the focal loss with a contiguity regularization term. To attain top performing results, a set of additional steps are proposed to ensure a reliable, generic, robust and efficient algorithm. First, we leverage our prior knowledge on the images by filling the holes enclosed by temporarily-classified blade pixels and by the image boundaries. Subsequently, the mislead classified pixels are successfully amended by training an on-the-fly random forest. Our algorithm demonstrates its effectiveness reaching a non-trivial 97.39% of accuracy.
BibTeX
@inproceedings{perezgonzalo2024robust,
  author = {Raül Pérez-Gonzalo and Andreas Espersen and Antonio Agudo},
  title = {{Robust wind turbine blade segmentation from RGB images in the wild}},
  booktitle = {ICIP},
  year = {2024},
  url = {https://openreview.net/forum?id=iqMCncYTy6},
}
Figure