Token Pruning for In-Context Generation in Diffusion Transformers

Junqing Lin, Xingyu Zheng, Pei Cheng, Bin Fu, Jingwei Sun, Guangzhong SunarXiv 2026 · arXiv · OpenReview

TL;DR

Diffusion Transformers (DiTs) are powerful for in-context image generation, but concatenating reference images dramatically increases computational cost. This paper introduces ToPi, a training-free method that intelligently prunes redundant reference tokens. ToPi achieves over 30% speedup in inference while preserving image quality, making in-context generation more efficient.

What Problem Does This Paper Address?

In-context generation with Diffusion Transformers (DiTs) allows for highly controllable image synthesis by conditioning on reference examples. However, this approach involves concatenating reference image tokens with the target image tokens, leading to significantly longer sequences. This increased sequence length creates a substantial computational bottleneck, especially due to the quadratic complexity of self-attention mechanisms in transformers. Existing token reduction techniques, often designed for text-based tasks, fail to adequately address the unique challenges of in-context image generation, where the relevance of reference tokens changes dynamically throughout the diffusion process. These methods tend to prune indiscriminately or retain too much redundancy, impacting generation fidelity.

Methods

ToPi tackles this problem with a three-pronged, training-free approach:

  1. Representative Layer Anchoring (Offline Calibration): Instead of analyzing all attention layers, ToPi first identifies a small subset of "representative layers" that are most sensitive to the context. This is done offline using a calibration dataset and a "Context Sensitivity Score" to pinpoint layers crucial for understanding the visual context. This reduces the computational burden of subsequent analyses.
  2. Attention-Weighted Importance Scoring (Periodic): To quantify the importance of each reference token, ToPi introduces a novel "Token Influence Score." This metric goes beyond just attention weights by also considering the magnitude of the token's feature vector. This ensures that tokens with moderate attention but strong features are retained, and vice-versa. This scoring is performed periodically, not at every step, to balance adaptivity with efficiency.
  3. Token Pruning & Context Realignment (Step-wise): Based on the calculated influence scores, ToPi adaptively prunes reference tokens. It uses a fidelity constraint (controlled by a threshold $\tau$) to ensure that a certain amount of information is preserved. This pruning is applied dynamically at each denoising step, and the remaining tokens are realigned to maintain structural integrity. A temporal update strategy ensures the pruning mask adapts to the evolving diffusion process.

Datasets

  • Calibration Set: A custom set of 48 image-text pairs, stratified across varying subtasks (e.g., Local Editing, Global Editing) was used for the offline calibration of representative layers.
  • Evaluation Benchmarks:
    • Flux.1-Kontext: Used for evaluating the framework on various editing tasks.
  • Qwen-Image-Edit: Another model used for evaluation, particularly on multi-image reference editing tasks.
  • AnyEdit: A benchmark used for evaluating performance across four sub-tasks: Camera Move Editing, Global Editing, Implicit Editing, and Local Editing.

Key Insights and Ideas

  • Asymmetric Roles of Tokens: The paper highlights that reference context tokens and target latent tokens play fundamentally different roles in in-context generation. Reference tokens act more as "auxiliary semantic anchors" rather than dense constructive elements, and their utility changes dynamically.
  • Pivotal Layers for Context: Not all layers in a DiT are equally important for processing context. Identifying and focusing on a few "representative layers" is sufficient for robust importance estimation and significantly reduces computation.
  • Beyond Attention Weights: Simply relying on attention weights is insufficient for pruning. The magnitude of token features (value vectors) also plays a crucial role in determining a token's semantic contribution.
  • Dynamic Pruning is Key: A fixed set of pruned tokens is suboptimal. The relevance of reference tokens evolves throughout the diffusion process, necessitating a temporal adaptation strategy for effective pruning.
  • Pruning vs. Merging: Selective pruning (keeping important tokens) is superior to merging less important tokens into their neighbors, as merging can lead to over-smoothing and loss of fine-grained details.

Key Results

  • Speedup: ToPi achieves over 30% speedup in inference (up to 1.33x on Qwen-Image-Edit) compared to full-context baselines.
  • Fidelity: Despite significant pruning, ToPi maintains structural fidelity and visual consistency, often outperforming baselines in PSNR, SSIM, and LPIPS metrics. For example, it achieved a remarkable gain of over 5 in PSNR for the Implicit Change task on Flux.1-Kontext.
  • Low Overhead: The additional computational cost introduced by ToPi's scoring and selection mechanisms is negligible, adding less than 1% overhead to the total inference time.
  • Orthogonality: ToPi is orthogonal to temporal acceleration techniques, allowing for combined speedups (e.g., 2.34x faster when combined with TeaCache).
  • Qualitative Improvements: Visualizations show that ToPi effectively preserves critical semantic anchors and fine-grained details, avoiding artifacts seen in baseline methods.

Limitations and Future Work

  • Limitations: The paper acknowledges that while ToPi significantly reduces redundancy, it still relies on a fixed pruning ratio ($\tau$) and periodic updates. The "Reference-Only" baseline variants show that even manual constraints don't match ToPi's dedicated mechanism.
  • Future Work: The authors suggest that future work could explore more sophisticated adaptive mechanisms for pruning and context selection, potentially leading to even greater efficiency gains without compromising quality. The framework's success also opens avenues for exploring input-adaptive generative models more broadly.

Summary updated

Abstract

In-context generation significantly enhances Diffusion Transformers (DiTs) by enabling controllable image-to-image generation through reference examples. However, the resulting input concatenation drastically increases sequence length, creating a substantial computational bottleneck. Existing token reduction techniques, primarily tailored for text-to-image synthesis, fall short in this paradigm as they apply uniform reduction strategies, overlooking the inherent role asymmetry between reference contexts and target latents across spatial, temporal, and functional dimensions. To bridge this gap, we introduce ToPi, a training-free token pruning framework tailored for in-context generation in DiTs. Specifically, ToPi utilizes offline calibration-driven sensitivity analysis to identify pivotal attention layers, serving as a robust proxy for redundancy estimation. Leveraging these layers, we derive a novel influence metric to quantify the contribution of each context token for selective pruning, coupled with a temporal update strategy that adapts to the evolving diffusion trajectory. Empirical evaluations demonstrate that ToPi can achieve over 30\% speedup in inference while maintaining structural fidelity and visual consistency across complex image generation tasks.
BibTeX
@misc{lin2026token,
  author = {Junqing Lin and Xingyu Zheng and Pei Cheng and Bin Fu and Jingwei Sun and Guangzhong Sun},
  title = {{Token Pruning for In-Context Generation in Diffusion Transformers}},
  year = {2026},
  eprint = {2602.01609},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01609},
}

Loading related content...

Figure