TL;DR
CLiS (Certified Loop-invariant Stabilization) is a training-free, plug-and-play optimization that significantly speeds up video diffusion models on TPUs. By replacing the costly, repeated numerical stabilization steps in standard attention with a "certified" fixed stabilizer, it achieves up to a 19.6% reduction in end-to-end generation latency without any loss in video quality.
What Problem Does This Paper Address?
High-resolution video generation relies on Diffusion Transformers (DiTs) that process incredibly long sequences—often over 100,000 tokens for just a few seconds of video. In these models, the attention mechanism is the primary bottleneck.
While researchers usually focus on speeding up matrix multiplications (matmuls), this paper identifies an emerging hardware bottleneck on modern TPUs: the softmax operation. Specifically, "online softmax" (a standard technique to keep numbers from exploding during computation) requires repeated maximum reductions and rescaling within the inner loop of the attention kernel. On TPU architectures, these operations are surprisingly expensive because they compete for limited vector registers and have lower throughput than matmuls. As sequence lengths grow, this "non-matmul" overhead dominates the total cycle cost.
Methods
The authors propose CLiS, which moves the stabilization work out of the performance-critical inner loop.
- Loop-Invariant Stabilization: Instead of finding the maximum value in every iteration (online softmax), CLiS calculates a fixed "stabilizer" for each row before entering the main loop. This removes the need for repeated max-reductions and rescaling.
- Numerical Certification: Using the Cauchy-Schwarz inequality and basic statistics of the queries and keys, CLiS derives a safe upper bound for the attention scores. It then checks if this fixed stabilizer will keep the resulting exponentials within the hardware's representable range (preventing overflow to infinity or underflow to zero).
- Safe Fallback: If a specific attention head fails this "certification" check, the system automatically falls back to standard, robust online softmax. This ensures that the optimization never breaks the model's numerical integrity.
- Implementation: CLiS is implemented using Pallas (a JAX-based kernel language for TPUs) and integrates into existing fused-attention kernels without requiring any changes to model weights or architecture.
Datasets
The researchers evaluated CLiS across several domains:
- Video Diffusion: Tested on state-of-the-art models Wan 2.2 and LTX-2 at various resolutions (480p to 1440p) and sequence lengths (17K to 184K tokens).
- Video Quality: Evaluated using the VBench suite to ensure no degradation in visual quality.
- LLM Training: Preliminary experiments were conducted on causal LLMs using datasets like C4, ClimbMix, FineWeb-Edu, UltraChat, and PG-19.
Key Insights and Ideas
- Softmax is the new bottleneck: As hardware accelerators make matmuls faster, the "simple" parts of attention like exponentiation and reduction are becoming the new critical path.
- Register Pressure Matters: A major source of CLiS's speedup comes from reducing "spill/fill" traffic—the costly moving of data between fast registers and slower memory. By simplifying the inner loop, CLiS frees up registers, roughly halving this memory traffic.
- Certification vs. Heuristics: Unlike prior GPU-focused methods that use empirical "best guesses" for stabilization, CLiS uses a formal proof of safety. This makes it reliable enough for both inference and training.
- Training-Free Acceleration: Because it preserves the exact arithmetic of attention (when certified), it can be dropped into pre-trained models without any fine-tuning.
Key Results
- Inference Speedup: CLiS reduced attention kernel latency by up to 24.0% and overall DiT-step latency by up to 19.6% on TPU v6e and TPU7x.
- Zero Quality Loss: VBench scores confirmed that videos generated with CLiS are visually indistinguishable from those generated with standard softmax.
- Training Gains: In LLM experiments, CLiS improved training throughput by up to 13.5% for long-context (256K) models, suggesting its utility extends far beyond just video diffusion.
- Robustness: In a 40-step video generation run, over 98% of attention heads were successfully certified, with the remaining few safely handled by the fallback path.
Limitations and Future Work
- Certification Failures: While rare, some heads do fail certification. The authors note that models without "QK-Norm" (a specific architectural feature) can see certification rates drop during early stages of training from random initialization.
- Hardware Specificity: The current implementation is highly optimized for TPU architectures. While the mathematical principles are general, the specific performance gains from reducing register pressure are most pronounced on TPUs.
- Future Directions: The authors suggest exploring more aggressive stabilization for the few heads that currently fail certification and extending the method to other non-matmul bottlenecks in transformer architectures.
Summary updated