TL;DR
FADE (Fractional Anomalous Dynamics Extrapolation) is a training-free framework that accelerates Diffusion Transformers (DiTs) by up to 5x with minimal loss in generation quality. By modeling latent feature evolution as "anomalous diffusion" and using fractional calculus (the Mittag-Leffler operator), FADE captures complex, high-curvature trajectories that standard linear forecasting methods miss.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) are the current state-of-the-art for generating high-quality images, videos, and audio. However, their iterative sampling process is computationally expensive, requiring dozens of sequential forward passes through massive models.
To speed this up, researchers often use feature caching, where they skip full computations for certain layers or timesteps and instead "forecast" what those features should be based on previous steps. The problem is that current forecasting methods typically use integer-order operators (like standard Taylor expansions). These assume that features move in a relatively smooth, "Brownian" fashion.
The authors of FADE argue that DiT latent trajectories are actually non-stationary and highly curved. At large skip intervals, standard linear or polynomial forecasting accumulates errors rapidly, leading to "structural collapse" in the generated output.
Methods
FADE introduces a more flexible way to predict feature evolution by drawing on fractional calculus.
- Anomalous Diffusion Modeling: The authors demonstrate that the mean squared displacement (MSD) of DiT features follows a power-law scaling rather than the linear scaling characteristic of standard Brownian motion. This suggests the features are undergoing "anomalous diffusion."
- The Mittag-Leffler Operator: Instead of the standard exponential shift operator used in Taylor-based forecasting, FADE employs a Mittag-Leffler fractional operator. This introduces a fractional order parameter ($\gamma$) that allows the model to adapt to the specific curvature and "memory" of the latent trajectory at different stages.
- Stage-Aware Calibration: A key insight is that while these dynamics are complex, they are remarkably consistent across different input samples for a given layer and timestep. FADE uses a small calibration set (only 10 samples) to offline-profile the optimal fractional order $\gamma$ for every layer and denoising step.
- Look-Up Table (LUT) Retrieval: These optimal $\gamma$ values are stored in a multi-dimensional LUT. During online inference, the model simply retrieves the pre-calculated $\gamma$ in $O(1)$ time, adding virtually zero latency overhead while significantly improving forecasting accuracy.
Datasets
The authors evaluated FADE across three distinct modalities and architectures:
- Class-Conditional Image Generation: DiT-XL/2 evaluated on ImageNet-1K.
- Text-to-Image Generation: FLUX.1-dev evaluated on DrawBench.
- Text-to-Video Generation: HunyuanVideo (HunyuanLarge architecture) evaluated using the VBench suite.
- Calibration: MS-COCO (for FLUX.1) and T2V-CompBench (for HunyuanVideo) were used for the small-scale offline profiling.
Key Insights and Ideas
- The "U-Shaped" Pattern: In DiT-XL/2, the authors found a distinct temporal pattern: the middle denoising phase is more "sub-diffusive" (lower $\gamma$), while the early and late phases are more "super-diffusive." This explains why static, one-size-fits-all forecasting fails.
- Implicit Regularization Trap: Standard Taylor methods act as a rigid mathematical constraint that "smooths out" necessary trajectory spikes. While this might look stable locally, it destroys the high-frequency semantics needed for complex generation. FADE’s fractional orders allow the model to "follow the spikes" faithfully.
- Cross-Sample Stability: The most surprising finding is that the optimal way to predict a feature's movement depends more on its location in the network and the current denoising step than on the actual image being generated. This is what makes the offline LUT approach so effective.
Key Results
FADE consistently outperforms existing training-free baselines like TaylorSeer and HiCache:
- DiT-XL/2: At a caching interval of $N=4$, FADE achieved an FID of 2.41, outperforming the 15-step DDIM baseline (FID 5.09) and other caching methods (FID ~2.5-2.6) at the same computational cost.
- FLUX.1: FADE maintained high semantic alignment (CLIP score) and image reward even under aggressive acceleration, where other methods began to show blurred textures or altered text rendering.
- Efficiency: The LUT retrieval adds less than 1ms to the total generation time, making the acceleration gains almost entirely "free" in terms of overhead.
Limitations and Future Work
- Manual Calibration: The current method relies on a one-time offline calibration stage. While very fast (50s to 2000s depending on the model), it still requires a representative (albeit small) dataset.
- Operator Choice: The Mittag-Leffler operator was chosen for its simplicity and generalization properties. The authors suggest that other fractional operators or more complex parameterizations of $\gamma$ could further refine the quality-efficiency trade-off.
- Future Directions: The team plans to explore how these fractional dynamics might be used to improve the training of diffusion models themselves, rather than just accelerating inference.
Summary updated