DiffCollage: Parallel Generation of Large Content with Diffusion Models

Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, Ming-Yu LiuCVPR 2023 · OpenReview

Abstract

We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tasks, including infinite image generation, panorama image generation, and long-duration text-guided motion generation. Extensive experimental results with a comparison to strong autoregressive baselines verify the effectiveness of our approach.

Introduction

Diffusion models have revolutionized generative AI, yet they face significant scaling challenges. Training a model to generate high-resolution panoramas or long-duration animations requires specialized, high-volume datasets that are often unavailable or prohibitively expensive. The standard industry workaround—autoregressive generation—involves generating small segments sequentially and using the edge of one segment to "outpaint" the next.

However, autoregressive methods suffer from three primary flaws: sequential slowness, error accumulation (where small mistakes "drift" and snowball over time), and a lack of global context. Because the process is unidirectional, the end of a sequence cannot influence the beginning, making it impossible to create seamless loops or globally consistent structures. DiffCollage addresses these issues by treating large-scale content generation as a parallelizable graph problem.

Methodology: The Factor Graph Approach

DiffCollage moves away from sequential outpainting toward a compositional framework based on a factor graph representation. In this model, large content is decomposed into a set of overlapping nodes. Each node represents a local piece of the content (a "factor"), while the overlapping regions act as "variable nodes."

Bethe Approximation and Score Aggregation

To define the joint probability of the entire large-scale content, the researchers utilize the Bethe approximation from classical probability theory. This allows the "score" (the gradient used to remove noise during the diffusion process) of the whole image to be approximated as the sum of the scores of individual pieces, minus the scores of the overlapping regions.

Parallel Sampling

Because the total score is calculated as a summation of local scores, all pieces can be denoised simultaneously. This parallelization means that, provided sufficient hardware resources, generating a 10,000-pixel image takes roughly the same amount of time as generating a 512-pixel one. This is a significant departure from the $O(N)$ time complexity of sequential methods.

Versatile Graph Topologies

By adjusting the graph's connectivity, DiffCollage can accommodate various content shapes:

  • Linear Chain: Used for infinite horizontal landscapes.
  • Cycle Graph: Used for looped animations or 360-degree panoramas, ensuring the "last" piece perfectly matches the "first."
  • Grid Graph: Used for generating high-resolution images of arbitrary height and width.

Experimental Evaluation

The researchers evaluated DiffCollage across multiple domains, including image and human motion generation. For image tasks, they utilized the Landscape High Quality (LHQ) and LSUN Tower datasets, finetuning models like GLIDE and eDiff-I. For motion, they used the HumanML3D dataset for text-to-motion generation.

The FID+ Metric

To better evaluate the quality of stitched content, the authors introduced FID+. While standard Fréchet Inception Distance (FID) measures general image quality, FID+ specifically penalizes discontinuities or "seams" at the boundaries where pieces are merged.

Key Results

  • Visual Quality: On the LHQ and Tower datasets, DiffCollage achieved an FID+ score of 16.43, significantly outperforming the 28.94 achieved by autoregressive baselines. This indicates much smoother transitions between patches.
  • Generation Speed: DiffCollage proved to be roughly 4x faster than the best autoregressive baselines in practical tests (6.47s vs 26.43s for long images).
  • Long-Duration Motion: The model successfully generated 24-second motion sequences—four times longer than the training data—while maintaining physical plausibility and adhering to complex multi-action prompts (e.g., "run, then kick, then skip rope").

Discussion and Insights

A key advantage of DiffCollage is its ability to maintain global consistency. In a cycle graph, the noise at the "end" of a panorama influences the noise at the "beginning" during every step of the diffusion process, forcing them to converge into a seamless loop. Furthermore, the framework is "zero-shot" capable, meaning it can wrap around pre-trained models without requiring them to be specifically trained on collages.

However, the method is not without limitations. It relies on a conditional independence assumption: if two pieces do not overlap, they are assumed to be independent given the pieces between them. This can occasionally lead to "semantic drift" in very large graphs (e.g., a long snake accidentally gaining a second head). Additionally, while parallelization saves time, it increases the VRAM footprint, as multiple diffusion instances must run concurrently.

Conclusion

DiffCollage represents a significant step forward in large-scale content generation. By shifting from sequential to parallel, graph-based composition, it eliminates the drift associated with autoregressive models while providing a massive speedup. Its ability to generate seamless, globally consistent content across images and motion makes it a versatile tool for the next generation of generative media.

Summary updated

Abstract

We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tasks, including infinite image generation, panorama image generation, and long-duration text-guided motion generation. Extensive experimental results with a comparison to strong autoregressive baselines verify the effectiveness of our approach.
BibTeX
@inproceedings{zhang2023diffcollage,
  author = {Qinsheng Zhang and Jiaming Song and Xun Huang and Yongxin Chen and Ming-Yu Liu},
  title = {{DiffCollage: Parallel Generation of Large Content with Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=JcaeOvvuVs},
}
Figure