D3-Cache: Accelerating Diffusion Models with Dynamic Mode Decomposition

Yanbo Li, An Xiang, Xiaoxin Hu, Feng Wang, Cheng-zhong Xu, Kejiang Ye, Xitong GaoarXiv 2026 · arXiv

TL;DR

D³-Cache is a training-free acceleration method for Diffusion Transformers that achieves 2.6–3.8× speedups with minimal loss in quality. By applying Dynamic Mode Decomposition (DMD) to forecast network outputs and using an adaptive, "horizon-aware" gate to skip evaluations, it eliminates the need for per-model offline calibration and adds virtually no memory overhead.

What Problem Does This Paper Address?

Diffusion Transformers (DiTs) are the state-of-the-art for high-fidelity image and video generation, but they are computationally expensive. Generating a single sample requires dozens of iterative denoising steps, each involving a full forward pass through a multi-billion parameter model.

Existing "caching" methods try to skip some of these evaluations by reusing previous outputs or forecasting them. However, they often suffer from three main drawbacks:

  1. High Memory Usage: Methods like TaylorSeer store derivative histories for every layer, which can double or triple VRAM requirements—a dealbreaker for large video models.
  2. Rigid Calibration: Many techniques require a "calibration curve" for every new model to decide when it's safe to skip a step.
  3. Open-Loop Decisions: Most skip criteria don't account for the actual error of the forecast in real-time; they just follow a pre-set schedule.

Methods

The authors introduce D³-Cache (Dynamic Mode Decomposition Diffusion Cache), which shifts the focus from caching internal features to forecasting the final "velocity" (the network's output that the scheduler uses to update the latent state).

Key Components:

  • DMD Forecasting: Instead of simple linear extrapolation, D³-Cache fits a Dynamic Mode Decomposition model to a short history (typically just the last 3 steps) of exact velocities. DMD identifies the underlying temporal "modes" of the denoising trajectory, allowing for more accurate forecasts of how the image or video will evolve.
  • Horizon-Aware Gate: This is a closed-loop control system. Every time the model performs a full evaluation, it compares the result to the previous forecast. This "realized error" is used to estimate the risk of skipping future steps. If the estimated risk is below a tolerance threshold ($\tau$), the model skips the next evaluation and uses the DMD forecast instead.
  • Negligible Memory: Because it only stores a few snapshots of the final latent-sized velocity—rather than every layer's activations—the memory overhead is nearly zero.
Overview of D³-Cache architecture and the adaptive sampling loop

Datasets

The researchers evaluated D³-Cache across four state-of-the-art models:

  • Images: FLUX.1-dev and HiDream-I1-full, tested on DrawBench (200 prompts) and COCO val2017 (100 captions).
  • Videos: Wan2.1-T2V-1.3B and HunyuanVideo, tested on VBench (128 prompts).

Key Insights and Ideas

  • Velocity is Highly Structured: The authors found that patch-level velocity updates in DiTs are strongly correlated. This structure is what makes a low-rank method like DMD so effective at forecasting (see Figure 1 in the paper).
  • Forecasting > Reuse: Simple "reuse" (using the output from step $t-1$ for step $t$) works okay, but it fails to capture the curvature of the denoising trajectory. DMD-based forecasting stays much closer to the "ground truth" 50-step path.
  • One Setting to Rule Them All: Remarkably, the authors used the same history length ($H=3$) and risk tolerance ($\tau=0.56$) across all four models (image and video), suggesting that the dynamics of diffusion models are consistent enough for a universal configuration.

Key Results

D³-Cache consistently outperformed established baselines like TeaCache and TaylorSeer in both speed and fidelity.

  • Speedup: Achieved 2.6× to 3.8× acceleration.
  • Fidelity: On the HiDream model, D³-Cache reached a 2.59× speedup while maintaining significantly higher SSIM (0.855 vs 0.772 for the next best) and lower LPIPS (0.111 vs 0.189) compared to MagCache.
  • Video Performance: On HunyuanVideo, D³-Cache was the only method that could provide high-order forecasting without exceeding GPU memory. It improved PSNR by over 3 dB compared to first-order baselines at matched speeds.
  • Qualitative Quality: Qualitative results show that while other methods might blur details or change the background during acceleration, D³-Cache preserves the layout, colors, and fine textures of the original 50-step output.

Limitations and Future Work

  • Warmup Steps: The method requires a few initial "warmup" steps (4 for images, 10 for videos) to build a history before it can start forecasting.
  • Solver Compatibility: While it works excellently with Euler and UniPC solvers, the paper notes that the gate doesn't yet provide formal error bounds for more complex multistep solvers.
  • Local Linearity: DMD assumes the trajectory is locally linear in a higher-dimensional space. While this holds well for diffusion, extremely "curvy" trajectories might still require more frequent refreshes.

The authors suggest that future work could involve integrating D³-Cache with distillation techniques or exploring even more sophisticated dynamical systems models for the forecasting step.

Summary updated

Abstract

Diffusion Transformers generate high-quality images and videos, but every sampling step requires a full network evaluation. Training-free caching reduces this cost by reusing or forecasting network outputs across steps, but common approaches use per-model calibrated skip criteria or maintain per-layer prediction histories. We present D3-Cache, which applies Dynamic Mode Decomposition (DMD) to Diffusion sampling: it fits DMD to a short history of exact velocities and forecasts only the velocity that the scheduler consumes. A horizon-aware gate decides when to skip: whenever the network is evaluated, its output is compared with the forecast for that step, and the larger the error, the sooner the network is evaluated again, up to a cap on consecutive skipped steps. D3-Cache requires no training or per-model offline calibration curve and adds negligible memory. It shares one history length and risk tolerance across the four evaluated models. On four state-of-the-art image and video models, D3-Cache achieves speedups of 2.6-3.8x with high fidelity to the base model output.
BibTeX
@misc{li2026d,
  author = {Yanbo Li and An Xiang and Xiaoxin Hu and Feng Wang and Cheng-zhong Xu and Kejiang Ye and Xitong Gao},
  title = {{D3-Cache: Accelerating Diffusion Models with Dynamic Mode Decomposition}},
  year = {2026},
  eprint = {2611.25689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25689},
}
Figure