TL;DR
ManiFusion is a novel framework that increases diffusion model throughput by processing multiple samples simultaneously within a single denoising forward pass. By leveraging the low-dimensional nature of image manifolds and a dynamic "Trajectory-Fusion Scheduling" strategy, it achieves nearly 2x throughput gains with minimal quality loss, effectively stacking with existing acceleration methods.
What Problem Does This Paper Address?
Diffusion models are the gold standard for high-fidelity image generation, but they are notoriously slow due to their iterative nature. Most research to date has focused on reducing latency—the time it takes to generate a single image—by reducing the number of denoising steps or caching intermediate features.
However, gains in per-sample latency are hitting a point of diminishing returns. The authors of ManiFusion argue for an orthogonal approach: increasing throughput. Instead of trying to make one forward pass faster, why not make one forward pass do more work? The challenge lies in "superposition": how do you mix multiple distinct image trajectories into a single shared representation without them becoming a garbled mess of pixels?
Methods
ManiFusion introduces a backbone-agnostic framework for multi-sample generation through fused-space computation.
- Fusion and Unfusion: The system takes $K$ independent noise inputs and fuses them into a single representation using index-conditioned encoding and summation. This fused representation is processed by the "Body Network" (the heavy lifting part of the U-Net or Transformer). Afterward, an "Unfusion" module separates the processed features back into $K$ distinct outputs.
- Trajectory-Fusion Scheduling (TFS): This is the paper's secret sauce. The authors observed that the early stages of diffusion (coarse structure) are more robust to fusion than the later stages (fine detail). TFS dynamically adjusts the fusion factor $K$ throughout the sampling process—for example, processing 4 samples at once during the noisy start and gradually reducing to 1 sample for the final refinement.
- Architectural Flexibility: The authors demonstrate ManiFusion across three main paradigms:
- MF-UNet: Splitting pixel-space U-Nets at the final convolutional layer.
- MF-DiT: Splitting Diffusion Transformers at the final linear projection.
- MF-AE: Applying fusion at the representation level of a latent autoencoder.
Datasets
The framework was evaluated across a diverse set of standard benchmarks:
- CIFAR-10: For both unconditional and class-conditional generation.
- CelebA & CelebA-HQ: For face generation.
- ImageNet: To test scalability on high-resolution, complex conditional generation using Diffusion Transformers (DiT).
Key Insights and Ideas
- The Manifold Hypothesis as a Buffer: The core "aha moment" is geometric. Because natural images live on low-dimensional manifolds ($d$) embedded in high-dimensional ambient space ($D$), there is mathematically "room" to pack multiple samples ($Kd$) into one representation, provided $D \ge Kd$.
- Throughput is Orthogonal to Latency: Because ManiFusion changes how many samples are processed rather than how they are processed, it can be combined with other methods. You can use ManiFusion alongside step-reduction (like DDIM) and feature caching (like DeepCache) to reach throughput regimes that were previously impossible.
- Conditional Diversity Helps: Interestingly, the authors found that mixing different classes within a fused group actually makes it easier for the model to separate them during unfusion, as the distinct class signals act as anchors for the de-multiplexing process.
Key Results
ManiFusion consistently achieves nearly 2x throughput (images per second) across various backbones and samplers with negligible impact on image quality metrics like FID (Fréchet Inception Distance).
As shown in Figure 2, ManiFusion significantly shifts the Pareto front of the quality-throughput trade-off. In high-throughput regimes (e.g., 500+ img/s on CIFAR-10), ManiFusion maintains much lower FID scores compared to vanilla models pushed to the same speed via step reduction alone.
Limitations and Future Work
- Training Requirement: Unlike some "plug-and-play" acceleration methods, ManiFusion currently requires full pretraining to teach the fusion/unfusion modules how to handle the shared space. This makes it computationally expensive to adapt to existing massive models.
- Scaling Limits: While $K=2$ and $K=4$ work well, extreme fusion factors (like $K=8$) lead to noticeable quality degradation, suggesting there is a hard limit to how much information can be "superposed" before the manifold becomes too crowded.
- Foundation Models: The study focuses on research-scale models (DiT-S, LDM). Testing this on massive text-to-image foundation models (like Stable Diffusion 3) remains an open and exciting challenge.
Summary updated