Structure-Guided Adversarial Training of Diffusion Models

Ling Yang, Haotian Qian, Zhilong Zhang, Jingwei Liu, Bin Cui

Abstract

Standard diffusion models are typically trained using instance-level denoising, which focuses on recovering individual samples in isolation. This approach overlooks the rich structural information present in the relationships between samples within a training batch. We introduce Structure-guided Adversarial training of Diffusion Models (SADM), a novel framework that forces the model to learn the underlying manifold structures of the data distribution. By employing a comparison-based structure discriminator in a minimax game, SADM ensures that the generated data distribution aligns with the authentic geometric relationships of the real data. Our method achieves state-of-the-art performance, including a record-breaking FID of 1.58 on ImageNet 256x256, and demonstrates superior generalization across multiple downstream tasks.

Introduction

In the current landscape of generative modeling, diffusion models have emerged as a dominant paradigm. However, most existing frameworks treat training samples as isolated entities. During the denoising process, the model attempts to map a noisy input back to its clean counterpart without considering how that input relates to others in the same mini-batch. This "instance-level" optimization ignores the fact that data points exist on a complex manifold where pairwise relationships—such as similarity and distance—define the true nature of the distribution.

By neglecting these structural relationships, models often struggle to perfectly align their learned manifold with the actual data distribution, leading to sub-optimal image quality and slower convergence. SADM addresses this gap by shifting the training focus from individual pixels to batch-level geometry.

Methodology

Structural Constraints and Affinity Matrices

SADM moves beyond pixel-wise loss by introducing a structural constraint. For every training batch, the model computes an affinity matrix that captures the pairwise relationships (using metrics like Euclidean distance or cosine similarity) between samples in a low-dimensional feature space. This matrix serves as a mathematical representation of the batch's geometric structure.

The Structure Discriminator

At the heart of SADM is a "comparison-based" structure discriminator. Unlike traditional GAN discriminators that evaluate single images, this discriminator analyzes the entire batch's structure. It projects both real and generated samples into an embedding space to identify a manifold where the differences between real and synthetic structures are most pronounced.

Adversarial Minimax Game

The training process is framed as a minimax game between the diffusion generator (the denoiser) and the structure discriminator. The discriminator aims to maximize the structural discrepancy between real and generated batches, while the denoiser works to minimize it. This adversarial pressure prevents the model from settling for easily learned, superficial structures and forces it to capture the authentic, complex manifold of the training data. Theoretically, this can be viewed as a "joint sample diffusion" process, where the model denoises a set of samples while simultaneously being conditioned on their mutual relations.

Experimental Results

Performance Benchmarks

SADM was evaluated across 12 diverse datasets, ranging from standard benchmarks like CIFAR-10 and CelebA to large-scale challenges like ImageNet. The model achieved state-of-the-art (SOTA) results, notably reaching an FID of 1.58 on ImageNet 256x256 and 2.11 on ImageNet 512x512. These figures outperform previous leading models such as the Masked Diffusion Transformer (MDT) and DiT-XL.

Qualitative and Generalization Insights

Heatmap analysis of the learned affinities reveals that SADM’s internal correlations are significantly closer to ground-truth label affinities than those of standard DDPMs. Furthermore, the model exhibits faster convergence than standard Diffusion Transformers (DiT), as it gains a more holistic view of the data distribution early in the training process.

In cross-domain fine-tuning tasks involving eight downstream datasets (including Food101, Caltech101, and Oxford Flowers), SADM achieved an average FID of 14.15. This significantly outperforms standard DDPM (16.59) and parameter-efficient methods like DiffFit, suggesting that learning manifold structures enhances the model's ability to adapt to new domains.

Conclusion and Future Work

SADM demonstrates that the relationships between images are as vital as the images themselves for high-fidelity synthesis. By incorporating adversarial structure-guided training, the model achieves superior image quality and faster convergence without the typical instabilities associated with GANs. While the calculation of pairwise affinity matrices introduces some computational overhead during training, the inference time remains unchanged. Future research will explore extending this structure-guided approach to text-to-image and text-to-video generation, where maintaining structural consistency across complex prompts and temporal frames remains a significant challenge.

Summary updated

Abstract

Diffusion models have demonstrated exceptional efficacy in various generative applications. While existing models focus on minimizing a weighted sum of denoising score matching losses for data distribution modeling, their training primarily emphasizes instance-level optimization, overlooking valuable structural information within each mini-batch, indicative of pair-wise relationships among samples. To address this limitation, we introduce Structure-guided Adversarial training of Diffusion Models (SADM). In this pioneering approach, we compel the model to learn manifold structures between samples in each training batch. To ensure the model captures authentic manifold structures in the data distribution, we advocate adversarial training of the diffusion generator against a novel structure discriminator in a minimax game, distinguishing real manifold structures from the generated ones. SADM substantially improves existing diffusion transformers (DiT) and outperforms existing methods in image generation and cross-domain fine-tuning tasks across 12 datasets, establishing a new state-of-the-art FID of 1.58 and 2.11 on ImageNet for class-conditional image generation at resolutions of 256x256 and 512x512, respectively.
BibTeX
@inproceedings{yang2024structure,
  author = {Ling Yang and Haotian Qian and Zhilong Zhang and Jingwei Liu and Bin Cui},
  title = {{Structure-Guided Adversarial Training of Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure