TL;DR
D³-Cache is a training-free acceleration method for Diffusion Transformers that achieves 2.6–3.8× speedups with minimal loss in quality. By applying Dynamic Mode Decomposition (DMD) to forecast network outputs and using an adaptive, "horizon-aware" gate to skip evaluations, it eliminates the need for per-model offline calibration and adds virtually no memory overhead.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) are the state-of-the-art for high-fidelity image and video generation, but they are computationally expensive. Generating a single sample requires dozens of iterative denoising steps, each involving a full forward pass through a multi-billion parameter model.
Existing "caching" methods try to skip some of these evaluations by reusing previous outputs or forecasting them. However, they often suffer from three main drawbacks:
- High Memory Usage: Methods like TaylorSeer store derivative histories for every layer, which can double or triple VRAM requirements—a dealbreaker for large video models.
- Rigid Calibration: Many techniques require a "calibration curve" for every new model to decide when it's safe to skip a step.
- Open-Loop Decisions: Most skip criteria don't account for the actual error of the forecast in real-time; they just follow a pre-set schedule.
Methods
The authors introduce D³-Cache (Dynamic Mode Decomposition Diffusion Cache), which shifts the focus from caching internal features to forecasting the final "velocity" (the network's output that the scheduler uses to update the latent state).
Key Components:
- DMD Forecasting: Instead of simple linear extrapolation, D³-Cache fits a Dynamic Mode Decomposition model to a short history (typically just the last 3 steps) of exact velocities. DMD identifies the underlying temporal "modes" of the denoising trajectory, allowing for more accurate forecasts of how the image or video will evolve.
- Horizon-Aware Gate: This is a closed-loop control system. Every time the model performs a full evaluation, it compares the result to the previous forecast. This "realized error" is used to estimate the risk of skipping future steps. If the estimated risk is below a tolerance threshold ($\tau$), the model skips the next evaluation and uses the DMD forecast instead.
- Negligible Memory: Because it only stores a few snapshots of the final latent-sized velocity—rather than every layer's activations—the memory overhead is nearly zero.
Datasets
The researchers evaluated D³-Cache across four state-of-the-art models:
- Images: FLUX.1-dev and HiDream-I1-full, tested on DrawBench (200 prompts) and COCO val2017 (100 captions).
- Videos: Wan2.1-T2V-1.3B and HunyuanVideo, tested on VBench (128 prompts).
Key Insights and Ideas
- Velocity is Highly Structured: The authors found that patch-level velocity updates in DiTs are strongly correlated. This structure is what makes a low-rank method like DMD so effective at forecasting (see Figure 1 in the paper).
- Forecasting > Reuse: Simple "reuse" (using the output from step $t-1$ for step $t$) works okay, but it fails to capture the curvature of the denoising trajectory. DMD-based forecasting stays much closer to the "ground truth" 50-step path.
- One Setting to Rule Them All: Remarkably, the authors used the same history length ($H=3$) and risk tolerance ($\tau=0.56$) across all four models (image and video), suggesting that the dynamics of diffusion models are consistent enough for a universal configuration.
Key Results
D³-Cache consistently outperformed established baselines like TeaCache and TaylorSeer in both speed and fidelity.
- Speedup: Achieved 2.6× to 3.8× acceleration.
- Fidelity: On the HiDream model, D³-Cache reached a 2.59× speedup while maintaining significantly higher SSIM (0.855 vs 0.772 for the next best) and lower LPIPS (0.111 vs 0.189) compared to MagCache.
- Video Performance: On HunyuanVideo, D³-Cache was the only method that could provide high-order forecasting without exceeding GPU memory. It improved PSNR by over 3 dB compared to first-order baselines at matched speeds.
- Qualitative Quality: Qualitative results show that while other methods might blur details or change the background during acceleration, D³-Cache preserves the layout, colors, and fine textures of the original 50-step output.
Limitations and Future Work
- Warmup Steps: The method requires a few initial "warmup" steps (4 for images, 10 for videos) to build a history before it can start forecasting.
- Solver Compatibility: While it works excellently with Euler and UniPC solvers, the paper notes that the gate doesn't yet provide formal error bounds for more complex multistep solvers.
- Local Linearity: DMD assumes the trajectory is locally linear in a higher-dimensional space. While this holds well for diffusion, extremely "curvy" trajectories might still require more frequent refreshes.
The authors suggest that future work could involve integrating D³-Cache with distillation techniques or exploring even more sophisticated dynamical systems models for the forecasting step.
Summary updated