TL;DR
This paper introduces an "importance-based token merging" technique for diffusion models that significantly improves generation quality and efficiency. By intelligently prioritizing and merging tokens based on their relevance to the input prompt, the method achieves state-of-the-art results in image and video generation while reducing computational costs.
What Problem Does This Paper Address?
Modern generative models, particularly diffusion models, rely on processing a large number of "tokens" (representing image patches or features) through complex neural networks. This process, especially the attention mechanisms, is computationally intensive and can be a bottleneck for generating high-quality images and videos. Previous attempts to speed this up by merging similar tokens often did so arbitrarily, leading to a loss of crucial details and structural integrity, especially in important regions of the generated content. The core problem is how to efficiently reduce the computational load without sacrificing generation quality, particularly by ensuring that the most semantically and structurally important information is preserved.
Methods
The paper proposes an "importance-based token merging" method. The key innovation lies in how tokens are selected for merging. Instead of random grouping, the method leverages "importance scores" for each token. A particularly effective and readily available source for these scores is Classifier-Free Guidance (CFG), which naturally highlights regions that strongly influence the model's output.
Here's a breakdown of their approach:
- Importance Scoring: They calculate an importance score for each token by taking the absolute difference between the conditional and unconditional noise predictions from the diffusion model (derived from CFG). This score reflects how much a token contributes to steering the generation towards the desired output.
- Important Token Pool: A pool of "important" tokens is created based on these scores.
- Destination and Source Tokens: Destination (dst) tokens, which will be the anchors for merging, are randomly sampled from this important token pool. The remaining tokens in the pool become source (src) tokens.
- Bipartite Soft Matching: A soft matching strategy is used to link src tokens to their most similar dst tokens.
- Independent Tokens: To avoid issues with tokens lacking suitable matches, a subset of src tokens that are least similar to any dst token are designated as "independent" tokens. This ensures that even if a token doesn't find a perfect match, it can still be processed.
- Merging: src tokens (and any remaining unimportant tokens) are merged into their corresponding dst tokens, typically by averaging their features.
- Efficient Computation: The diffusion model then processes this reduced set of tokens, and the merged features are used to fill the locations of the destination tokens, maintaining the original output shape.
This approach aims to balance computational efficiency with the preservation of critical visual details by intelligently allocating resources to important regions.
Datasets
The paper evaluates its method on standard datasets for various generative tasks:
- Text-to-Image Synthesis: MS-COCO 2014 validation dataset.
- Multi-view Diffusion: GSO dataset.
- Video Diffusion: VBench.
The paper does not introduce new benchmarks.
Key Insights and Ideas
- CFG as an Importance Signal: The paper highlights that Classifier-Free Guidance (CFG), a standard technique in diffusion models, can be effectively repurposed as a direct indicator of token importance without any additional computational cost during inference.
- Content-Aware Merging is Crucial: Arbitrary token merging (e.g., based purely on spatial proximity) degrades generation quality by discarding important details. Prioritizing tokens based on their relevance to the prompt is key to maintaining fidelity.
- Balancing Relevance and Diversity: Simply picking the top-k most important tokens as destinations can lead to redundancy. The proposed method's strategy of sampling from an important pool and designating "independent" tokens helps strike a balance between focusing on important areas and maintaining diversity.
- Generalizability: The importance-based token merging approach demonstrates effectiveness across various model architectures (U-Net, transformers) and generative tasks (text-to-image, multi-view, video).
Key Results
The proposed importance-based token merging method consistently outperforms baseline methods like ToMeSD and ToFu across various tasks and merging ratios.
- Text-to-Image: Achieved an FID score of 17.75 at a merging ratio of 0.75, compared to ToMeSD's 20.89. This indicates higher quality and more coherent generations.
- Multi-view Diffusion: Showed improved PSNR and SSIM scores, and significantly lower LPIPS, especially at higher merging ratios (e.g., 14.58 PSNR vs. 13.12 for ToMeSD at r=0.75), indicating better preservation of geometric and textual details.
- Video Diffusion: Consistently performed better than ToMeSD in both numerical metrics and visual quality.
- Efficiency: The method achieves significant speedups (e.g., 5s inference time vs. minutes for ATC) while maintaining or improving generation quality. The inference costs (TFLOPs, latency, GPU memory) are comparable to or better than baselines, demonstrating that efficiency gains don't come at the cost of increased resource usage.
- Ablation Studies: Showed that using top-k destination tokens without the proposed pool and independent token strategy leads to worse results. Using cross-attention maps as an alternative importance signal also yielded strong results, highlighting the method's flexibility.
Limitations and Future Work
- Consistency Models: While the method works well with standard diffusion models, its application to consistency models (which distill CFG) is noted as a potential challenge. The authors suggest using alternative importance signals like attention maps for these models.
- Multi-Guidance Scenarios: For scenarios with multiple guidance signals (e.g., InstructPix2Pix), the authors suggest exploring weighted averaging of importance signals based on user preferences.
- Refining Distillation: The authors propose future work on refining distillation processes to predict an additional output: a classifier-free guidance map, which could be used for better token merging or other innovative applications.
- Token Pruning in Early Steps: The paper explores combining token pruning in early diffusion steps with token merging, showing it can further preserve structural details. This suggests that a hybrid approach might be beneficial.
Summary updated