Temporal Redundancy and Feature Caching in Diffusion Transformers

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2270 papers429 conceptsfrozen 2026-10-04

Diffusion models and Diffusion Transformers (DiTs) have become the standard for high-fidelity generation, but their iterative nature makes them computationally expensive. To generate a single image or video, these models must pass through the same neural network dozens of times. However, recent research has revealed a critical inefficiency: the model’s internal representations—its attention maps, activations, and feature vectors—change remarkably little between consecutive denoising steps.

This phenomenon, known as temporal redundancy, suggests that much of the computation in diffusion inference is unnecessary. By strategically caching and reusing features from previous steps, researchers are achieving massive speedups without sacrificing quality.

The Intuition: The Latent Space Moves Slowly

In the early stages of a diffusion process, the model performs "semantic planning," establishing the high-level structure and layout of the image. In the final stages, it performs "refinement," adding fine-grained textures. Between these two phases lies a vast "middle ground" where the model’s trajectory is highly stable.

Wentian Zhang, Haozhe Liu, and others observed in Faster Diffusion via Temporal Attention Decomposition that cross-attention maps (which link text prompts to pixels) converge to a stable state very early. Once the model "knows" where the "dog" is in the frame, recomputing that relationship at every step is redundant. Similarly, Chaoyi Jiang and Sung Woo Kim found in MARché that token representations often maintain a cosine similarity above 0.95 between steps, meaning the GPU is essentially performing the same math over and over.

The U-Shaped Redundancy Pattern

The redundancy in diffusion is not uniform. Xuanlei Zhao identifies a "U-shaped" pattern in Real-Time Video Generation with Pyramid Attention Broadcast. Attention outputs change rapidly at the very beginning (when noise is first being shaped) and at the very end (when tiny details are finalized). However, during the middle 70% of the process, the outputs are nearly static.

This observation led to Pyramid Attention Broadcast (PAB), which reuses attention results across multiple steps during these stable periods. By skipping these redundant calculations, PAB enables real-time 720p video generation with up to a 10.5x speedup.

Caching Strategies: From Maps to Offsets

Researchers have developed several distinct ways to exploit this stability:

  1. Attention Map Reuse: Methods like TGATE (Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models) and HURRY (Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models) cache the high-dimensional products of queries and keys. Instead of recomputing the attention matrix, the model simply retrieves the cached version, drastically reducing the cost of the most expensive layers.
  2. Delta Caching: Rather than caching raw feature maps, Pengtao Chen and colleagues proposed $\Delta$-DiT ($\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers). This method stores the difference (the delta) between input and output features. This preserves the "information chain" of the transformer more effectively than simple skipping, allowing for faster sampling while actually improving image quality.
  3. Adaptive Block Skipping: Models like LazyDiT (LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers) and HarmoniCa (HarmoniCa: Harmonizing Training and Inference for Better Feature Cache in Diffusion Transformer Acceleration) use lightweight routing mechanisms to decide, on the fly, if a specific transformer block needs to be run or if its previous output is "good enough" to be reused.

Parallelism through "Stale" Activations

Temporal redundancy also unlocks new forms of hardware parallelism. In xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism, Jiarui Fang and Aoyu Li introduce PipeFusion.

Standard parallel processing often stalls because one GPU must wait for another to finish its calculation. PipeFusion breaks the image into patches and uses "stale" activations from the previous denoising step as context for the current step. Because the features change so slowly, using a slightly outdated activation from a neighboring patch doesn't hurt the final image quality, but it allows multiple GPUs to work simultaneously without waiting for communication, turning minutes of generation into seconds.

Beyond Pixels: Sparsity and Tokens

The same logic applies to the tokens themselves. In RedVTP, Jingqi Xu shows that the relative importance of visual tokens remains consistent across the entire diffusion process. This allows the model to prune "unimportant" tokens (like background pixels) early in the process and never look at them again. Similarly, SparseD (SparseD: Sparse Attention for Diffusion Language Models) by Gongfan Fang computes a sparse connectivity mask once and reuses it for every subsequent iteration, significantly reducing the overhead of sparse attention.

By treating the diffusion process not as a series of independent steps, but as a continuous, slowly-evolving trajectory, these methods are transforming diffusion from a heavy, offline process into a responsive, real-time technology.

Papers in this version

Showing 60 of 270 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 180 papers
  2. Version 1 · 2026-10-04 · 270 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_feature_caching_temporal_featur_6abfb682,
  title = {temporal feature caching (temporal feature redundancy)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-feature-caching-temporal-feature-redundancy/rdr_6abfb682127b41aa8712e7e4d6304ed5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_redundancy_and_feature_caching__6abfb682_v1,
  title = {Temporal Redundancy and Feature Caching in Diffusion Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-redundancy-and-feature-caching-in-diffusion-transformers/rdr_6abfb682127b41aa8712e7e4d6304ed5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}