Abstract
Training high-quality Denoising Diffusion Probabilistic Models (DDPMs) typically requires massive datasets and significant computational resources. This paper introduces SR-DDPM (Shared-Representation DDPM), a hierarchical approach designed for effective multi-task image generation in data-constrained environments. By partitioning a UNet architecture into shared core parameters and task-specific exclusive layers, SR-DDPM learns common structural features across categories while preserving unique task-specific details. Experimental results on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 demonstrate that SR-DDPM outperforms standard conditional diffusion models in both image quality (FID) and structural similarity (SSIM), even with as few as 500 samples per task.
Introduction
Denoising Diffusion Probabilistic Models (DDPMs) have become a gold standard for image generation, yet they suffer from a significant "data hunger" problem. Producing realistic results often requires thousands of samples per category, which is impractical in domains like medical imaging or niche artistic styles where data is scarce.
While conditional diffusion models can handle multiple tasks by taking class labels as input, they often struggle to balance shared underlying structures with the specific nuances of small datasets. This research bridges that gap by leveraging techniques from few-shot meta-learning—specifically the concept of shared versus personalized representations—to make diffusion models more data-efficient and scalable.
The SR-DDPM Framework
The authors propose SR-DDPM, which modifies the standard monolithic UNet architecture used in diffusion models into a hierarchical structure.
- Shared vs. Exclusive Layers: The architecture is split into shared parameters ($\phi$), representing the "meta" architecture, and task-specific parameters ($\theta_i$) reserved for each individual task $i$.
- Architectural Design: In experiments, the authors personalized the first and last layers of a 4-layer UNet as the "exclusive" stages. The middle layers remain shared, allowing the bottleneck to learn common structural features while the exclusive layers handle task-specific low-level features and high-level reconstructions.
- Training and Sampling: During training, the algorithm randomly samples a task and an image, performing a standard denoising step that updates both shared and task-specific parameters. For generation, the model simply plugs the corresponding exclusive parameters ($\theta_i$) into the shared core ($\phi$).
Experimental Setup
The model was evaluated on four standard benchmarks using a limited data regime of only 500 samples per task:
- MNIST: 10 tasks (handwritten digits).
- Fashion-MNIST (FMNIST): 10 tasks (clothing items).
- CIFAR-10: 10 tasks (general objects/animals).
- CIFAR-100: 20 tasks (finer-grained object categories).
Results and Analysis
SR-DDPM was compared against unconditional DDPM and Conditional DDPM (C-DDPM). The model consistently achieved superior performance across key metrics:
- FID (Fréchet Inception Distance): SR-DDPM achieved the lowest (best) FID scores. For example, on MNIST, SR-DDPM reached an FID of 2.04, compared to 2.14 for C-DDPM and 3.67 for standard DDPM.
- SSIM (Structural Similarity Index): The model outperformed or matched conditional models, particularly on complex datasets like CIFAR-10 (0.949) and CIFAR-100 (0.944).
- Visual Quality: Qualitative results demonstrate that even with limited data, the model produces clear, distinct images for each class without the blurriness typical of low-data regimes.
Key Insights
- Implicit Similarity Detection: The exclusive layers automatically detect similarities between tasks. For instance, the model identified overlaps between "T-shirts" and "Dresses" in FMNIST, suggesting the shared layers successfully captured the general "clothing-ness" of the data.
- Fine-Tuning Efficiency: Because the core is shared, adding new tasks does not require full retraining. New exclusive layers can be trained while keeping shared weights frozen, enabling faster and lighter fine-tuning.
Conclusion
SR-DDPM demonstrates that hierarchical representation sharing significantly improves the performance of diffusion models in low-data scenarios. While the current work uses empirical testing to determine layer splits, future research could explore automated methods like neural architecture search to optimize the balance between shared and exclusive parameters. Additionally, scaling the number of tasks beyond 20 remains an open area for investigation.
Summary updated