Free Lunch of Image-mask Alignment for Anomaly Image Generation and Segmentation

Xiangyue Li, Xiaoyang Wang, Zhibin Wan, Quan Zhang, Yupei Wu, Tao Deng, Mingjie SunIJCAI 2025 · DOI

Abstract

Training deep learning models for anomaly detection in specialized domains like industrial quality control and medical imaging is often hindered by the scarcity of real-world anomalous data. While synthetic data generation offers a potential solution, existing generative models frequently suffer from "label drift," where generated anomalies do not perfectly align with their corresponding segmentation masks. This paper introduces a novel framework that addresses this misalignment through a dual-branch training strategy and leverages the generative model's internal representations to refine segmentation performance. By enforcing pixel-level alignment and utilizing a "Generative Feedback" mechanism, the proposed method achieves significant performance gains across multiple benchmarks, including a 16.6% IoU increase in indoor anomaly detection.

Introduction

In fields such as medical diagnostics and industrial manufacturing, identifying rare defects or lesions is critical. Synthetic data generation using diffusion models has become a popular approach to augment training sets. However, a persistent challenge is the lack of precise spatial correspondence between the generated image and the input mask. When a segmentation model is trained on such misaligned data, it learns incorrect features, leading to poor generalization on real-world images. This research aims to bridge this gap by ensuring perfect image-mask alignment and then using the generative model as a "teacher" to provide feedback to the segmentation "student."

Methodology

The proposed framework, typically implemented using Stable Diffusion and Segformer, consists of two primary stages: Dual-Branch Generative Training and Generative Feedback.

Dual-Branch Generative Training

Unlike standard conditional generation where the mask is merely an input, this approach treats the mask as a target. A second branch is added to the diffusion model to denoise the mask alongside the image. To ensure pixel-level consistency, an Alignment Regularization ($L_{al}$) loss is introduced. This loss minimizes the discrepancy between the noise predicted for the image and the noise predicted for the mask, forcing the model to align features at the pixel level.

Overview of the proposed training pipeline, including the dual-branch generator training and the generative feedback loop for segmentation refinement.

Generative Feedback

Once the generative model is trained, it is used to fine-tune the segmentation model. During this stage, the segmentation model's predicted mask is fed into the frozen generative model. The generative model's denoising loss serves as a Generative Feedback ($L_{fb}$) signal. If the predicted mask is accurate, the generative model can more effectively reconstruct the image, resulting in a lower loss. This mechanism allows the segmentation model to benefit from the generative model's deep understanding of the image-mask relationship without requiring additional human labels.

Experimental Results

The method was evaluated on diverse datasets, including polyp segmentation (medical), Real-IAD and MVTec-AD (industrial), and Floor Dirty (indoor).

Performance Gains

The proposed approach consistently outperformed state-of-the-art (SOTA) methods. In industrial tasks (Real-IAD), the mean Intersection over Union (mIoU) improved by 5.03%. In medical imaging, a 5.68% mIoU increase was observed. The most significant improvement occurred in the Floor Dirty dataset, where mIoU rose by 16.63%.

Qualitative and Quantitative Quality

Quantitative metrics such as the Fréchet Inception Distance (FID) confirmed that the synthetic images were significantly more realistic than those from previous models like ArSDM (162.01 vs. 361.55). Qualitative results demonstrate that the generated anomalies are not only realistic but also perfectly aligned with the target masks across various domains.

Qualitative comparison of generated anomalies across industrial, medical, and floor datasets, showing superior alignment and realism compared to baseline methods.

Conclusion

This work demonstrates that a well-trained diffusion model contains a "free lunch" of knowledge that can be extracted for segmentation tasks. By treating masks as targets rather than just conditions and implementing a generative feedback loop, the proposed method effectively solves the label drift problem and sets new performance benchmarks. While the computational cost of the feedback loop is a consideration, the significant gains in accuracy and the potential for universal application to other conditional tasks make this a promising direction for future research.

Summary updated

BibTeX
@inproceedings{li2025free,
  author = {Xiangyue Li and Xiaoyang Wang and Zhibin Wan and Quan Zhang and Yupei Wu and Tao Deng and Mingjie Sun},
  title = {{Free Lunch of Image-mask Alignment for Anomaly Image Generation and Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/162},
}

Loading related content...

Figure