Controllable Image Synthesis of Industrial Data Using Stable Diffusion

Gabriele Valvano, Antonino Agostino, Giovanni De Magistris, Antonino Graziano, Giacomo VeneriWACV 2024

TL;DR

This paper tackles the challenge of limited annotated data in industrial settings by proposing a novel method to adapt pre-trained diffusion models like Stable Diffusion for generating synthetic, defect-containing industrial images. This approach allows for the creation of self-annotated datasets, significantly boosting the performance of downstream tasks like crack segmentation, especially when real data is scarce.

What Problem Does This Paper Address?

Training robust computer vision models for industrial applications, such as defect detection and segmentation, typically requires vast amounts of precisely annotated data. However, acquiring such datasets in industrial environments is often prohibitively expensive and time-consuming. Defects, by their nature, are rare, and obtaining detailed segmentation masks for them is a significant hurdle. While generative AI offers a promising avenue for data augmentation, even training state-of-the-art generative models from scratch demands substantial data. This paper addresses this critical data scarcity problem by leveraging powerful, pre-trained foundation models and adapting them to the specific domain of industrial data, enabling the generation of realistic, defect-annotated images without needing extensive real-world data.

Methods

The proposed method consists of two main phases, building upon a pre-trained Stable Diffusion model:

  1. Phase 1: Learn the Concept: This phase adapts the pre-trained diffusion model to understand the specific domain of industrial images, which often differ significantly from the natural images Stable Diffusion was originally trained on (as illustrated in Figure 2). The authors employ a technique similar to DreamBooth, fine-tuning the model with a small subset of labeled industrial data. This process injects knowledge of the new industrial "concept" while preserving the model's general image generation capabilities. A class-specific prior preservation loss is used to prevent catastrophic forgetting.
  2. Phase 2: Learn the Condition: In this phase, the model is trained to generate images that adhere to specific constraints, such as defect geometry and location. This is achieved by using a HyperNetwork that conditions the pre-trained diffusion model. The conditioning input is a "topological driver," which is an RGB image containing coarse topological information of the scene and a defect mask. The HyperNetwork learns to guide the diffusion process to produce images that match this topological driver and the specified defect characteristics. The text prompt is also slightly modified to include the defect class.

Datasets

  • Turbine Boroscope Inspection (TBI) Dataset: This is an in-house dataset containing RGB borescope images of gas turbines, compressors, and combustion chambers. It includes both defect-free images and images with defects (cracks). Crucially, the defective images come with instance-level segmentation masks for each crack, categorized into five classes. For training, a subset of 1170 annotated images was used, with the remaining data split for validation and testing of instance segmentation models. Defect-free images from this dataset were also used for synthesis during inference.

Key Insights and Ideas

  • Bridging the Domain Gap: Pre-trained foundation models like Stable Diffusion possess a powerful general image prior, but this prior doesn't directly align with industrial data distributions. A dedicated "concept learning" phase is essential to bridge this domain gap.
  • Conditional Generation with Topological Drivers: The use of a HyperNetwork to condition the diffusion model with a topological driver (combining coarse scene geometry and defect masks) is a novel way to control the generation process for specific industrial defects.
  • Self-Annotated Data for Downstream Tasks: The generated synthetic images come with accurate, known segmentation masks by construction. This "self-annotation" capability is crucial for training downstream models when real annotations are scarce.
  • Inpainting vs. Full Synthesis: The paper argues that simple inpainting (filling defects into existing images) is insufficient for industrial data due to its limited mode coverage. Their approach, which generates entire images, offers greater diversity.

Key Results

  • Improved Instance Segmentation: When using synthetic data generated by their method to train a Mask RCNN model for crack segmentation on the TBI dataset, performance significantly improved across multiple metrics (Segm mAP, BBox mAP, IoU, HD) compared to training on real data alone, especially when only 10% of the real data was available.
  • Stabilized Training with Limited Data: For challenging setups with only 10% of the annotated data, training a Mask RCNN model directly led to collapse. However, augmenting this small dataset with synthetic data stabilized training and achieved performance close to using the entire real dataset.
  • Realistic and Diverse Synthesis: The generated synthetic images were qualitatively realistic and showed defects with geometries and positions coherent with the input masks. The Frechet Inception Distance (FID) for the generated data was 96.57, indicating good realism.
  • Better Mode Coverage than Inpainting: A comparison with an inpainting-based method showed that the proposed approach achieved better mode coverage and data generation quality.

Limitations and Future Work

  • Data Leakage Prevention: The authors intentionally held out defect-free samples for inference and synthesis, using only labeled data for training the concept and condition learning phases to prevent data leakage.
  • Future Work: The paper suggests exploring other strategies for generating defect masks for synthesis, such as using another neural network. They also mention investigating other concept learning approaches like LoRA and Textual Inversion as future work. The current method relies on a fixed set of defect classes and geometries, and extending this to a wider range of defects or more complex topological conditions could be a future direction.

Summary updated

BibTeX
@inproceedings{valvano2024controllable,
  author = {Gabriele Valvano and Antonino Agostino and Giovanni De Magistris and Antonino Graziano and Giacomo Veneri},
  title = {{Controllable Image Synthesis of Industrial Data Using Stable Diffusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure