TL;DR
ToMA (Token Merge with Attention) is a training-free, plug-and-play method designed to accelerate diffusion transformers like SDXL and Flux without degrading image quality. By replacing GPU-inefficient operations like sorting and scattered writes with submodular optimization and attention-like linear transformations, ToMA delivers a practical 23–24% reduction in generation latency.
What Problem Does This Paper Address?
State-of-the-art diffusion models have increasingly shifted toward transformer backbones (e.g., DiT, U-ViT, Flux) to achieve high-fidelity image generation. However, the quadratic computational complexity of self-attention scales poorly with token counts, creating a massive latency bottleneck during the iterative denoising process.
While prior works attempted to solve this via token reduction (merging or pruning redundant tokens), they suffer from a fundamental hardware mismatch. Methods like ToMeSD and ToFu rely on GPU-unfriendly operations such as sorting and scattered memory writes. In the modern inference stack—where the attention mechanism itself is already highly optimized using kernels like FlashAttention—the overhead introduced by these unoptimized token-merging steps actually dwarfs the theoretical computational savings, sometimes making models slower than the baseline. ToMA bridges this gap between theoretical FLOP reduction and real-world wall-clock speedups.
Methods
ToMA redesigns token reduction from the ground up to align with modern GPU execution paradigms:
- Submodular-Based Destination Selection: Instead of heuristic matching, ToMA models token selection as a submodular optimization problem using a Facility Location function. This mathematically guarantees the selection of a diverse and highly representative subset of "destination" tokens to preserve global semantic structure.
- Merge/Unmerge via Attention: ToMA reformulates the merging operation as a soft assignment using standard Scaled Dot-Product Attention (SDPA) kernels, mapping $N$ tokens to $D$ destinations. For the crucial "unmerge" step required to restore spatial consistency in generative tasks, ToMA utilizes the matrix transpose of the attention weights. The authors prove that due to the diversity of the selected tokens and a sharp attention temperature, the merge matrix is nearly orthonormal, making the cheap transpose an excellent mathematical approximation of the expensive Moore-Penrose pseudo-inverse.
- Locality and Redundancy Exploitation: To minimize overhead, ToMA processes tokens within localized 2D tiles or horizontal stripes, allowing for massive parallelization. Furthermore, it exploits sequential redundancy by reusing the selected destination tokens (every 10 steps) and the computed merge weights (every 5 steps) across consecutive layers and denoising timesteps.
Datasets
The authors evaluated ToMA using prompts drawn from two primary sources:
- GEMRec dataset: Used for evaluating text-to-image generation performance.
- ImageNet-1K: Class names were used as prompts to generate a benchmark of 3,000 images to compute standard distribution metrics against ground-truth data.
Key Insights and Ideas
- The FlashAttention Paradox: Optimizing the primary bottleneck (attention) shifts the critical path to the auxiliary logic (token management). If your compression logic isn't hardware-aligned, it will bottleneck the optimized system.
- Leveraging the SDPA "Free Ride": By casting token merging as an attention operation, ToMA automatically inherits the hardware efficiencies and future optimizations applied to standard attention kernels.
- The Orthonormality Shortcut: A key mathematical insight is that a well-chosen, diverse set of destination tokens combined with low-temperature softmax yields a nearly orthonormal assignment matrix. This allows the system to completely bypass costly matrix decompositions (like SVD or QR) during the unmerge phase, substituting them with a simple, lightning-fast matrix transpose.
Key Results
ToMA was benchmarked against competitive baselines (ToMe, ToFu, ToDo) across multiple GPU architectures (NVIDIA RTX6000, V100, RTX8000).
- Real-World Speedups: ToMA achieved a 24% latency reduction on SDXL-base and a 23% latency reduction on Flux.1-dev, outperforming prior methods.
- Quality Preservation: The acceleration comes with negligible image degradation, maintaining a DINO score change of less than 0.07.
- Baseline Comparison: While ToMA consistently accelerated generation, the popular ToMeSD baseline actually caused a 42-43% slowdown when paired with FlashAttention2 due to its GPU-inefficient sorting mechanisms.
Limitations and Future Work
- Memory Copying Overhead: Although ToMA's locality-aware tiling and striping strategies drastically reduce computational complexity, they still incur minor GPU memory reshaping and copying overhead. The authors note that developing custom tiled/stripe attention kernels could eliminate this overhead entirely.
- RoPE Incompatibility with Stripes: For DiT architectures like Flux that utilize Rotary Positional Embeddings (RoPE), the faster "stripe-shaped" region partitioning is incompatible. Flux is therefore restricted to the slightly slower "tile-shaped" variant to maintain geometric consistency.
Summary updated