Matryoshka Diffusion Models

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, Navdeep Jaitly

Generating high-fidelity $1024 \times 1024$ images or videos is often considered the "final boss" of generative AI. While diffusion models have become the industry standard, scaling them to high resolutions usually requires complex workarounds. Developers typically resort to training a "cascade" of separate models (one for $64 \times 64$, another for $256 \times 256$, etc.) or moving to a compressed "latent space" (as seen in Stable Diffusion).

In a new paper, researchers from Apple introduce Matryoshka Diffusion Models (MDM). Named after the famous Russian nesting dolls, MDM suggests a more elegant path: training a single pixel-space model that learns to denoise multiple resolutions simultaneously by nesting the features of small scales within the larger ones.

A collage of high-resolution images generated by Matryoshka Diffusion Models, demonstrating diverse capabilities.

TL;DR

Matryoshka Diffusion Models (MDM) enable high-resolution image and video synthesis within a single pixel-space model by jointly denoising multiple resolutions. By using a NestedUNet architecture and a progressive training schedule, MDM achieves faster convergence and high-quality $1024 \times 1024$ generation without the need for separate cascaded models or lossy latent auto-encoders.

The Problem with High-Resolution Diffusion

High-resolution diffusion is computationally expensive and notoriously difficult to optimize. Standard UNet architectures struggle to capture global structure and fine details simultaneously when trained directly on high-res pixels.

To solve this, current state-of-the-art methods typically use:

  1. Cascaded Diffusion: Training multiple independent models that progressively upsample images. This pipeline is complex to manage and prone to "exposure bias," where errors in the low-res stage amplify in the high-res stage.
  2. Latent Diffusion (LDM): Training on a downsampled representation from a pre-trained auto-encoder. While highly efficient, the final image quality is strictly bounded by the (often lossy) compression of that auto-encoder.

MDM asks a fundamental question: Can we exploit the hierarchical nature of images to train a single, efficient, high-resolution model directly in pixel space?

The Solution: Nested Learning

The core of MDM is the NestedUNet. In a traditional UNet, the model processes the input at a single resolution. In a NestedUNet, the features and parameters used to process a $64 \times 64$ version of the image are literally a subset of the features used for the $256 \times 256$ version, which are in turn nested within the $1024 \times 1024$ version.

MDM generates images at multiple nested resolutions simultaneously, from 64x64 up to 1024x1024.

Key technical components of this approach include:

  • Multi-resolution Diffusion: The model is trained to denoise a vector of inputs representing the same image at different resolutions (e.g., 64, 256, 1024) simultaneously.
  • Parameter Sharing: Low-resolution inputs are fed into an "inner" UNet. The resulting features are then passed to "outer" layers that handle higher resolutions. This ensures that the high-res layers don't have to "re-learn" the basic structural composition already captured by the low-res layers.
  • Progressive Training: The authors don't start training at $1024 \times 1024$. They begin by training the inner-most (low-res) blocks and gradually "grow" the model by adding higher-resolution stages. This acts as a natural curriculum, stabilizing the optimization process significantly.

Key Results and Insights

The authors evaluated MDM across several standard benchmarks, including ImageNet ($256 \times 256$), Conceptual 12M (CC12M), WebVid-10M, and MS-COCO. The results highlight several major advantages:

  • Faster Convergence: MDM converges significantly faster than a standard UNet trained on high-res images. On ImageNet, MDM reached a lower FID (better quality) in half the training iterations of baseline models.
  • State-of-the-Art Quality: On ImageNet $256 \times 256$, MDM achieved an impressive FID of 3.51, outperforming standard ADM (10.94) and even Latent Diffusion (3.60).
  • Zero-Shot Generalization: Despite being trained on the relatively small CC12M dataset (only 12 million image-text pairs), MDM produced high-quality $1024 \times 1024$ images that generalized well to complex prompts, outperforming much larger models like DALL-E 2 on MS-COCO benchmarks.
  • Video Synthesis: The model successfully extended to video generation (16 frames at $256 \times 256$), proving that the Matryoshka principle works across temporal dimensions as well.

By nesting resolutions, the model allocates the bulk of its parameters and computation to the lowest resolution, where global structure is decided. The high-resolution "shells" add relatively few parameters, making the jump from 256 to 1024 surprisingly cheap. Ultimately, MDM proves that we don't have to use latent spaces to get high-res results, avoiding the artifacts sometimes introduced by VAE-based compression.

Limitations and Future Work

While MDM is highly efficient, the authors note that it has not yet been tested at the "foundation model" scale (e.g., billions of parameters trained on billions of images). Additionally, the current default weights the loss of different resolutions based on their pixel count; more sophisticated weighting schemes could further improve performance.

Finally, the authors acknowledge that MDM and Latent Diffusion are not mutually exclusive. A future "Matryoshka Latent Diffusion" model could potentially combine the best of both worlds—nesting resolutions within a latent space for extreme efficiency.

Summary updated

Abstract

Diffusion models are the de-facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges. Existing methods often resort to training cascaded models in pixel space, or using a downsampled latent space of a separately trained auto-encoder. In this paper, we introduce Matryoshka Diffusion (MDM), an end-to-end framework for high-resolution image and video synthesis. We propose a diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small-scale inputs are nested within those of large scales. In addition, MDM enables a progressive training schedule from lower to higher resolutions, which leads to significant improvements in optimization for high-resolution generation. We demonstrate the effectiveness of our approach on various benchmarks, including class-conditioned image generation, high-resolution text-to-image, and text-to-video applications. Remarkably, we can train a single pixel-space model at resolutions of up to 1024x1024 pixels, demonstrating strong zero-shot generalization using the CC12M dataset, which contains only 12 million images. Code and pre-trained checkpoints are released at https://github.com/apple/ml-mdm.
BibTeX
@inproceedings{gu2024matryoshka,
  author = {Jiatao Gu and Shuangfei Zhai and Yizhe Zhang and Joshua M. Susskind and Navdeep Jaitly},
  title = {{Matryoshka Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=tOzCcDdH9O},
}
Figure