Abstract
Training deep learning models for anomaly detection in specialized domains like industrial quality control and medical imaging is often hindered by the scarcity of real-world anomalous data. While synthetic data generation offers a potential solution, existing generative models frequently suffer from "label drift," where generated anomalies do not perfectly align with their corresponding segmentation masks. This paper introduces a novel framework that addresses this misalignment through a dual-branch training strategy and leverages the generative model's internal representations to refine segmentation performance. By enforcing pixel-level alignment and utilizing a "Generative Feedback" mechanism, the proposed method achieves significant performance gains across multiple benchmarks, including a 16.6% IoU increase in indoor anomaly detection.
Introduction
In fields such as medical diagnostics and industrial manufacturing, identifying rare defects or lesions is critical. Synthetic data generation using diffusion models has become a popular approach to augment training sets. However, a persistent challenge is the lack of precise spatial correspondence between the generated image and the input mask. When a segmentation model is trained on such misaligned data, it learns incorrect features, leading to poor generalization on real-world images. This research aims to bridge this gap by ensuring perfect image-mask alignment and then using the generative model as a "teacher" to provide feedback to the segmentation "student."
Methodology
The proposed framework, typically implemented using Stable Diffusion and Segformer, consists of two primary stages: Dual-Branch Generative Training and Generative Feedback.
Dual-Branch Generative Training
Unlike standard conditional generation where the mask is merely an input, this approach treats the mask as a target. A second branch is added to the diffusion model to denoise the mask alongside the image. To ensure pixel-level consistency, an Alignment Regularization ($L_{al}$) loss is introduced. This loss minimizes the discrepancy between the noise predicted for the image and the noise predicted for the mask, forcing the model to align features at the pixel level.
Generative Feedback
Once the generative model is trained, it is used to fine-tune the segmentation model. During this stage, the segmentation model's predicted mask is fed into the frozen generative model. The generative model's denoising loss serves as a Generative Feedback ($L_{fb}$) signal. If the predicted mask is accurate, the generative model can more effectively reconstruct the image, resulting in a lower loss. This mechanism allows the segmentation model to benefit from the generative model's deep understanding of the image-mask relationship without requiring additional human labels.
Experimental Results
The method was evaluated on diverse datasets, including polyp segmentation (medical), Real-IAD and MVTec-AD (industrial), and Floor Dirty (indoor).
Performance Gains
The proposed approach consistently outperformed state-of-the-art (SOTA) methods. In industrial tasks (Real-IAD), the mean Intersection over Union (mIoU) improved by 5.03%. In medical imaging, a 5.68% mIoU increase was observed. The most significant improvement occurred in the Floor Dirty dataset, where mIoU rose by 16.63%.
Qualitative and Quantitative Quality
Quantitative metrics such as the Fréchet Inception Distance (FID) confirmed that the synthetic images were significantly more realistic than those from previous models like ArSDM (162.01 vs. 361.55). Qualitative results demonstrate that the generated anomalies are not only realistic but also perfectly aligned with the target masks across various domains.
Conclusion
This work demonstrates that a well-trained diffusion model contains a "free lunch" of knowledge that can be extracted for segmentation tasks. By treating masks as targets rather than just conditions and implementing a generative feedback loop, the proposed method effectively solves the label drift problem and sets new performance benchmarks. While the computational cost of the feedback loop is a consideration, the significant gains in accuracy and the potential for universal application to other conditional tasks make this a promising direction for future research.
Summary updated