Dual-Space Augmented Intrinsic-LoRA for Wind Turbine Segmentation

Shubh Singhal, Raül Pérez-Gonzalo, Andreas Espersen, Antonio AgudoarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel "dual-space augmentation" strategy for fine-tuning large pre-trained diffusion models (like Stable Diffusion) for the specific task of wind turbine blade (WTB) segmentation. By combining image-level interpolation with latent-space noise injection, their method, SI-LORA, significantly improves segmentation accuracy, outperforming existing state-of-the-art approaches.

What Problem Does This Paper Address?

Accurate segmentation of wind turbine blades (WTBs) is crucial for automated damage detection and maintenance. While large, general-purpose vision models (like SAM) show promise, they often struggle with domain-specific tasks like WTB segmentation due to a lack of specialized training data. Traditional fine-tuning methods can be data-hungry. This paper aims to bridge this gap by adapting pre-trained generative models using a technique called Intrinsic-LoRA, which allows for effective supervised learning with minimal labeled data, specifically for WTB segmentation.

Methods

The core of the proposed method is an adaptation of Intrinsic-LoRA for image segmentation. Intrinsic-LoRA leverages pre-trained diffusion models by introducing small, trainable "LoRA adaptors" that fine-tune the model for specific tasks without altering the original weights.

The authors extend this to segmentation by:

  1. Adapting Output: Modifying the model to produce a single-channel segmentation mask instead of a 3-channel image.
  2. Dual-Space Augmentation (DSA): This is the key innovation. It combines two augmentation strategies:
  3. Image-Space Augmentation: Uses MixUp [16], a technique that creates new training samples by linearly interpolating pairs of images and their corresponding masks.
  4. Latent-Space Augmentation: Introduces a noise-based probabilistic model for the latent representations within the diffusion model's U-Net. This involves treating the latent vector as a Bayesian input, modeled as an isotropic Gaussian distribution, and then reparameterizing it with added noise. This helps the model generalize better by exposing it to a wider range of latent inputs.

Datasets

  • Training Data: 1712 images.
  • Evaluation Data: 320 images.
  • Source: The dataset is taken from reference [4], which focuses on wind turbine blade segmentation from RGB images in the wild. The paper notes that the dataset includes images from various windfarms, captured with different drone configurations and locations, which is used to test the model's robustness.

Key Insights and Ideas

  • Domain-Specific Fine-Tuning is Key: Large universal models, while powerful, are not inherently optimal for specialized tasks like WTB segmentation. Fine-tuning pre-trained generative models with techniques like LoRA offers a more efficient path to high performance.
  • Latent Space Augmentation is Powerful: Simply augmenting images is effective, but augmenting the model's internal latent representations (where the core processing happens) provides a significant boost. This suggests that making the model robust to variations in its internal feature space is critical for generalization.
  • Synergy of Augmentations: The combination of image-level and latent-level augmentations yields a synergistic effect, leading to performance gains that are greater than the sum of their individual contributions.
  • Probabilistic Latent Space: Treating latent vectors probabilistically (as opposed to deterministically) makes the model more robust to variations and improves its ability to handle diverse inputs.

Key Results

  • Quantitative Performance: The proposed DSA SI-LORA achieves state-of-the-art results, with an accuracy of 99.15%, F1-score of 98.84%, and mIoU of 97.69% on their test set. This represents a significant improvement over the baseline SI-LORA (82.30% accuracy) and other competing models.
  • Ablation Study:
    • No augmentation: 82.30% Accuracy, 76.48% F1, 76.15% mIoU.
  • Latent-space augmentation only: 95.22% Accuracy, 93.33% F1, 91.76% mIoU (a 20.48% improvement in mIoU over baseline).
  • Image-space augmentation only: 97.59% Accuracy, 97.23% F1, 95.24% mIoU.
  • Dual-space augmentation: 99.15% Accuracy, 98.84% F1, 97.69% mIoU.
  • Qualitative Performance: Visual comparisons show that the dual-space augmented model produces much smoother and more complete segmentation masks, accurately capturing all parts of the wind turbine, including darker regions and internal structures, which simpler augmentation strategies or the base SI-LORA struggle with.
  • Robustness: The model demonstrates consistent high performance across 10 distinct windfarms, indicating its robustness to variations in capture conditions and environments.

Limitations and Future Work

  • Outliers: While generally robust, the boxplot analysis revealed a few outliers with lower performance on specific windfarms (WF 1, 3, and 4). Figure 5 illustrates some failure cases where high contrast between the WTB and background, or within the WTB itself, leads to misclassifications.
  • Future Work: The paper implicitly suggests that further refinement of the augmentation strategies or model architecture could address these outlier cases. The success of fine-tuning pre-trained generative models for specialized segmentation tasks also opens avenues for applying this approach to other domain-specific segmentation problems.

Summary updated

Abstract

Accurate segmentation of wind turbine blade (WTB) images is critical for effective assessments, as it directly influences the performance of automated damage detection systems. Despite advancements in large universal vision models, these models often underperform in domain-specific tasks like WTB segmentation. To address this, we extend Intrinsic LoRA for image segmentation, and propose a novel dual-space augmentation strategy that integrates both image-level and latent-space augmentations. The image-space augmentation is achieved through linear interpolation between image pairs, while the latent-space augmentation is accomplished by introducing a noise-based latent probabilistic model. Our approach significantly boosts segmentation accuracy, surpassing current state-of-the-art methods in WTB image segmentation.
BibTeX
@misc{singhal2024dual,
  author = {Shubh Singhal and Raül Pérez-Gonzalo and Andreas Espersen and Antonio Agudo},
  title = {{Dual-Space Augmented Intrinsic-LoRA for Wind Turbine Segmentation}},
  year = {2024},
  eprint = {2412.20838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20838},
}
Figure