IdealCache: Rethinking Cache Scheduling in Diffusion Transformers via Ideal Trajectories

Shang Xue, Li Zhu, Ping ChenarXiv 2026 · arXiv

TL;DR

IdealCache is a training-free framework that significantly accelerates Diffusion Transformers (DiTs) by optimizing when to cache intermediate features. By abandoning the goal of mimicking standard denoising paths and instead using graph-based optimization to target an "ideal" straight-line trajectory, it achieves up to 6.35x speedups with near-lossless quality, even outperforming standard pipelines in image editing consistency.

What Problem Does This Paper Address?

Diffusion Transformers (DiTs) are the backbone of modern high-fidelity image and video generation, but they are notoriously slow. Generating a single output requires dozens of iterative "denoising" steps, each involving an expensive forward pass through a massive transformer.

To solve this, researchers use feature caching: they run a full computation at one step and reuse those results for several subsequent steps. However, existing methods have a "blind spot." They try to make the cached version match the intermediate states of a standard 50-step "vanilla" pipeline as closely as possible. The authors argue that this is a mistake. The vanilla trajectory is just one arbitrary path to a result; by trying to imitate it, caching methods inherit that path's limitations as a "quality ceiling."

The challenge is: if we don't follow the standard path, how do we know which steps are the most important to compute fully?

Methods

The authors propose IdealCache, which shifts the focus from the journey (the denoising path) to the destination (the final image).

  1. The Ideal Trajectory: In "rectified flow" models, the most efficient path from random noise to a clean image is a straight line in latent space. IdealCache uses this straight line as its "anchor."
  2. Cache Deviation: The authors define a metric to measure how much "drift" is introduced when you reuse a cache over a specific interval. They calculate the distance between where the cached computation lands you and where the "ideal" straight-line trajectory says you should be. A visualization of cache deviation: the distance between a cached rollout and the ideal straight-line trajectory.

  3. Graph-Based Scheduling: They treat the entire denoising process as a Directed Acyclic Graph (DAG). Each node is a timestep, and each edge represents a caching interval with a "weight" equal to its cache deviation.

  4. Optimal Search: Finding the best schedule becomes a classic "constrained shortest path" problem. Given a budget of $B$ steps, the algorithm finds the specific sequence of steps that minimizes total deviation. This is solved exactly using dynamic programming—no training or fine-tuning required.
A visualization of cache deviation: the distance between a cached rollout and the ideal straight-line trajectory.

Datasets

The researchers evaluated IdealCache across three distinct modalities:

  • Text-to-Image: Evaluated on Qwen-Image using the DrawBench benchmark.
  • Image Editing: Evaluated on Qwen-Image-Edit using GEdit-Bench and FlowEdit.
  • Text-to-Video: Evaluated on HunyuanVideo using the VBench dataset.
  • Graph Construction: They used 1,000 prompts from the Pick-a-Pic dataset to build the "mean graph" used for optimization.

Key Insights and Ideas

  • Destination over Journey: The most profound insight is that the intermediate states of a diffusion model don't actually matter as long as the final output is high-quality. By decoupling from the vanilla trajectory, the model is "freed" to spend its limited compute budget where it helps the final result most.
  • Optimal Substructure: The authors prove that cache errors are "self-contained" within their intervals. This mathematical property allows them to use efficient shortest-path algorithms to find a globally optimal schedule rather than relying on hand-tuned heuristics.
  • The "Editing Bonus": In image editing, the "ideal" result is often the original image with a specific change. By anchoring the trajectory to the source image, IdealCache naturally penalizes "background drift," leading to edits that are more semantically consistent than the original 50-step model. Qualitative editing results showing superior background preservation compared to the vanilla 50-step baseline.

  • One-Shot Transferability: Surprisingly, a schedule optimized on just a single random prompt performs nearly as well as one optimized on a large ensemble. This makes the "offline" cost of finding a schedule almost negligible.

Qualitative editing results showing superior background preservation compared to the vanilla 50-step baseline.

Key Results

IdealCache consistently outperforms prior state-of-the-art caching methods (like HyCa and FORA) across the board:

  • Speed: It achieves 6.35x wall-clock speedup on Qwen-Image and 4.32x on HunyuanVideo.
  • Fidelity: At these high speeds, it maintains "near-lossless" quality. For example, on HunyuanVideo, it achieves significantly better LPIPS and SSIM scores than other caching baselines at similar compute budgets.
  • Image Editing: On the GEdit-EN benchmark, IdealCache (at 5.84x speedup) actually achieved a higher overall quality score (7.59) than the full 50-step vanilla pipeline (7.54). Representative outputs across image, editing, and video tasks showing high fidelity at high speedups.
Representative outputs across image, editing, and video tasks showing high fidelity at high speedups.

Limitations and Future Work

  • Base Model Dependency: IdealCache is a "pathfinder." If the underlying DiT model is incapable of generating a good image in 50 steps, IdealCache will simply find the fastest way to reach that same flawed result.
  • Extreme Compression: While it performs well down to very low budgets, the authors note that at extremely high compression (e.g., fewer than 5 full steps), residual errors eventually compound, leading to quality degradation.
  • Video Heterogeneity: While "one-shot" scheduling works for images, complex videos with high motion might still require an ensemble of reference runs to build a robust schedule.
  • Future Directions: The authors suggest that this framework could be adapted to future, even more powerful foundational models, as it is entirely architecture-agnostic.

Summary updated

Abstract

Feature caching accelerates Diffusion Transformers by reusing intermediate computations across denoising timesteps, yet existing methods share a fundamental blind spot: they treat the standard reference denoising trajectory as the optimisation target, approximating its per-step states and thus inheriting its output as an implicit quality ceiling. We argue that this target is arbitrary -- what truly matters is the ideal result, not the intermediate path taken to reach it. We propose IdealCache, a training-free framework that abandons step-wise imitation of the reference trajectory. Instead, it anchors caching on a per-input ensemble of ideal trajectories constructed from candidate target results, aggregates their geometry into a single graph, and discovers optimal non-uniform schedules via a constrained shortest path on this graph. By decoupling from the reference process, IdealCache breaks the performance bottleneck of trajectory-approximation; our schedules can attain near-lossless fidelity in generation and even surpass on task-specific metrics (e.g., maintaining higher semantic consistency in image editing). Without retraining, IdealCache achieves 6.35x speedup on Qwen-Image, 5.84x on Qwen-Image-Edit, and 4.32x on HunyuanVideo, with competitive or superior quality across tasks.
BibTeX
@misc{xue2026idealcache,
  author = {Shang Xue and Li Zhu and Ping Chen},
  title = {{IdealCache: Rethinking Cache Scheduling in Diffusion Transformers via Ideal Trajectories}},
  year = {2026},
  eprint = {2611.10340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10340},
}
Figure