TL;DR
Matryoshka Diffusion Models (MDM) introduce a novel, end-to-end approach for high-resolution image and video generation. By jointly denoising inputs across multiple resolutions within a single NestedUNet architecture and employing a progressive training schedule, MDM achieves impressive results, including 1024x1024 image generation from a relatively small dataset, without relying on cascaded or latent diffusion models.
What Problem Does This Paper Address?
Generating high-resolution images and videos with diffusion models is computationally demanding and optimization-challenging. Current state-of-the-art methods often resort to complex multi-stage pipelines: either training cascaded diffusion models (where a low-resolution model is followed by super-resolution models) or performing diffusion in a compressed latent space learned by a separate auto-encoder. These approaches introduce complexity in training and inference, and can be bottlenecked by the limitations of individual stages or the lossy compression of latent spaces. The authors aim to develop a more unified and efficient framework for high-resolution synthesis.
Methods
The core innovation is the Matryoshka Diffusion Model (MDM), which operates on an "extended space" where inputs are represented at multiple resolutions simultaneously.
- Joint Denoising Across Resolutions: Instead of separate models for different resolutions, MDM denoises inputs at multiple resolutions concurrently within a single diffusion process.
- NestedUNet Architecture: A specialized NestedUNet architecture is proposed. This architecture efficiently shares computation and parameters across resolutions, with features from lower resolutions being nested within those of higher resolutions. This allows for a more integrated processing of multi-scale information.
- Progressive Training Schedule: MDM employs a progressive training strategy. The model is first trained on lower resolutions, and then progressively higher resolutions are added to the training objective. This approach significantly speeds up convergence and improves optimization for high-resolution generation, avoiding the need for costly high-resolution training from the outset.
- Pixel-Space Generation: Notably, MDM can train a single model directly in pixel space up to 1024x1024 resolution, bypassing the need for latent diffusion or cascaded models.
Datasets
- ImageNet: Used for class-conditioned image generation (256x256 resolution).
- Conceptual 12M (CC12M): A dataset of approximately 12 million image-text pairs, used for general text-to-image generation at resolutions up to 1024x1024. The authors highlight CC12M's suitability for training high-quality models with good zero-shot capabilities, contrasting it with larger, less accessible datasets used in prior work.
- WebVid-10M: Used for text-to-video generation (16x256x256 resolution).
Key Insights and Ideas
- Hierarchical Structure Exploitation: The paper demonstrates that explicitly modeling the hierarchical structure of data (from low to high resolution) within a single diffusion process, rather than through separate stages, leads to significant improvements in efficiency and performance.
- Nested Architecture Efficiency: The NestedUNet architecture is key to enabling efficient multi-resolution processing. By nesting computations, it allows for substantial parameter sharing and computational reuse, making high-resolution generation more feasible.
- Progressive Training as a Shortcut: The progressive training schedule is not just a way to manage complexity but a crucial technique for accelerating convergence and achieving better results, especially for high-resolution tasks. It effectively "grows" the model's capability.
- CC12M as a Viable Benchmark: The successful training of high-resolution models on CC12M suggests that this dataset is a strong candidate for future benchmarks in text-to-image generation, offering a balance of scale and accessibility.
Key Results
- High-Resolution Generation: MDM successfully generates images at resolutions up to 1024x1024.
- Improved Convergence: The progressive training schedule significantly speeds up training and improves convergence compared to training directly at high resolutions.
- Competitive Performance: MDM achieves competitive FID and CLIP scores on benchmarks like ImageNet and CC12M, even when trained on smaller datasets than many prior methods. For instance, on CC12M 256x256, MDM achieves an FID of 3.51 with CFG.
- Zero-Shot Generalization: Models trained on CC12M demonstrate strong zero-shot capabilities for text-to-image generation.
- Video Generation: The framework is extended to video generation, showing promising results.
- Efficiency: Compared to cascaded diffusion models, MDM offers better performance with fewer parameters and inference steps, despite the latter having more combined parameters.
Limitations and Future Work
- Scalability: While MDM shows strong performance, the authors acknowledge that evaluating its full potential on even larger datasets and model sizes is a direction for future work.
- Complementarity with Latent Diffusion: The authors note that MDM and latent diffusion models are complementary, suggesting the possibility of building MDM on top of autoencoder codes.
- Architecture Exploration: Further exploration of weight sharing architectures and parameter distribution within the NestedUNet could yield additional improvements.
- Generalization to Other Domains: While demonstrated for images and videos, exploring MDM's applicability to other data modalities could be a future avenue.
Summary updated