As deep learning models scale to billions of parameters, deploying them efficiently becomes a massive engineering challenge. Traditional optimization techniques—such as knowledge distillation, network pruning, and quantization-aware training—typically require a complete retraining cycle. This "retraining overhead" creates a severe deployment bottleneck: practitioners often lack the massive compute resources, specialized hardware, or original training datasets required to fine-tune a frontier model.
To bypass this barrier, a powerful research direction has emerged: training-free, post-hoc optimization. Instead of modifying a model's weights via backpropagation, these methods exploit the inherent mathematical, spatial, and temporal redundancies already present within pre-trained weights. By treating the model as a frozen asset and optimizing its execution path at deploy-time, these techniques achieve dramatic speedups and memory reductions with near-zero performance degradation.
Exploiting Temporal and Layer-Wise Redundancy via Caching
The core intuition behind training-free caching is that deep learning computations often generate highly repetitive intermediate states. In iterative generation tasks, such as diffusion or flow matching, hidden states evolve smoothly across consecutive timesteps. Rather than recomputing these states from scratch at every step, we can cache and reuse them.
For Diffusion Transformers (DiTs), Joseph Liu 0001 et al. introduce SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers, which identifies and caches redundant computations across adjacent denoising timesteps using a rapid calibration pass. Similarly, Hanshuai Cui et al. develop BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching, which targets redundancy at the individual block level during the middle of the denoising process. For molecular geometry generation, Johanna Sommer et al. demonstrate in Predictive Feature Caching for Training-free Acceleration of Molecular Geometry Generation that predicting intermediate hidden states instead of recomputing them yields a 3x standalone speedup. This principle also applies to attention mechanisms; Rosco Hunter et al. show in Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models that reusing previously calculated attention maps preserves image fidelity better than simply reducing sampling steps.
Standard Iterative Step: [Input] -> [Compute Layer] -> [Output] (Repeated N times)
Cached Iterative Step: [Input] -> [Cached Layer] -> [Output] (Skips heavy compute)
This redundancy is not limited to the time domain; it also exists across the structural layers of Large Language Models (LLMs). Because hidden states remain highly similar from one layer to the next, Yixuan Wang et al. propose CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing, which shares Key-Value (KV) cache parameters across adjacent layers. To protect accuracy, Rongzhi Zhang et al. introduce LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy, applying low-rank approximations conservatively to shallow layers and aggressively to deeper layers to ease long-sequence inference bottlenecks without any retraining.
Spatial Redundancy and Token-Level Pruning
In vision and multimodal models, inputs frequently contain massive spatial redundancy—such as background pixels in an image or static frames in a video. Processing every single token through a deep network wastes significant computational cycles. Training-free token pruning exploits the model's internal attention weights to identify and eliminate or merge these uninformative tokens on-the-fly.
For Vision-Language Models (VLMs), Ahmadreza Jeddi proposes SAINT in Similarity-Aware Token Pruning: Your VLM but Faster, modeling tokens as nodes in a graph and utilizing a dynamic voting mechanism to prune redundant elements. Taking an attention-guided approach, Kazi Hasan Ibn Arif develops HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models, which drops visual tokens early before they reach the heavy LLM backbone.
Rather than completely discarding data, the authors of Video, How Do Your Tokens Merge? utilize Token Merging (ToMe) to mathematically combine redundant spatio-temporal patches in video transformers, achieving up to a 2.5x speedup. For convolutional and vision backbones, Tanvir Mahmud et al. introduce PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inference, while Pengyu Li et al. present Grid-Attention: Enhancing Computational Efficiency of Large Vision Models Without Fine-Tuning, which restructures transformer blocks to exploit spatial redundancy without requiring fine-tuning.
Zero-Shot Structural Conversions
Can we fundamentally alter a model's underlying mathematical operations to be more hardware-friendly without retraining? The answer lies in exact mathematical decompositions or structural approximations that preserve the model's input-output mappings.
In the convolutional domain, Jianbo Guo et al. prove in Network Decoupling: From Regular to Depthwise Separable Convolutions that standard convolutions can be exactly represented as a sum of depthwise separable convolutions (DSCs). This allows heavy, pre-trained CNNs to be converted into lightweight, MobileNet-like structures at deploy-time. For quantization, You Qiaoben et al. introduce Composite Binary Decomposition Networks, which decomposes full-precision weights into multi-bit binary networks without a lengthy training process.
For Transformers, the quadratic complexity ($O(N^2)$) of standard attention is a major bottleneck. Researchers have found ways to swap this mechanism for sub-quadratic alternatives zero-shot:
- Structured Matrices: Can Yaras et al. introduce MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention, which projects standard attention matrices into sub-quadratic ($O(N\sqrt{N})$) Monarch matrices without requiring retraining.
- Locality Pruning: Heejun Lee et al. present A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention, which exploits "attention locality" (the tendency for neighboring tokens to share importance) to prune attention maps into an $O(T \log T)$ structure.
- Online Filtering: Jintao Zhang et al. develop SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference, using a two-stage online filter to compress tokens dynamically during the forward pass.
- Algorithmic Reformulation: Yuzhen Mao et al. introduce IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs, which reformulates self-attention as a fast nearest-neighbor search problem to accelerate CPU inference.
Post-Hoc Routing and Behavioral Refinement
Beyond caching and pruning, training-free optimization can be used to dynamically route inputs or refine the geometric properties of a model's latent space.
Instead of executing every layer for every token, Ahmed Heakl et al. introduce Dr.LLM: Dynamic Layer Routing in LLMs, which retrofits frozen models with lightweight routers. These routers use paths discovered via Monte Carlo Tree Search to decide on-the-fly whether to skip, execute, or repeat a layer. For Mixture-of-Experts (MoE) architectures, Xudong Lu et al. demonstrate in Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models that redundant experts can be permanently pruned or temporarily bypassed post-training to halve GPU memory requirements.
Standard MoE: [Input] -> [Router] -> [Expert 1] + [Expert 2] + [Expert 3]
Pruned MoE: [Input] -> [Router] -> [Expert 1] (Redundant experts bypassed)
This post-hoc philosophy also extends to improving model robustness and representation quality. Chao Chen et al. show in Optimal Parameter and Neuron Pruning for Out-of-Distribution Detection that selectively pruning overconfident neurons using only the original training set statistics significantly improves out-of-distribution detection without retraining. To fix "representation degeneration"—where embeddings from cross-modal models like CLIP cluster into a narrow cone—Xiangru Jian et al. introduce InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution. This training-free post-processing technique applies an inverse graph convolution to push clustered representations apart, immediately boosting retrieval accuracy.
Finally, this modularity enables zero-shot architectural manipulation. The authors of Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation process retrieved documents in parallel paths and prune irrelevant ones to eliminate the quadratic cost of long-context attention zero-shot. On a macro scale, Jiacong Hu et al. introduce Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks, a framework that strips task-specific components from pre-trained models and snaps them together into new architectures, bypassing backpropagation entirely.
By shifting the optimization paradigm from weight modification to algorithmic execution design, training-free optimization effectively democratizes the deployment of large-scale models, allowing state-of-the-art architectures to run efficiently on commodity hardware.
Key Foundational Work
Refining Architectures of Deep Convolutional Neural Networks (2016) — contributed 1 concept elements to this direction.
Network Decoupling: From Regular to Depthwise Separable Convolutions (2018) — contributed 1 concept elements to this direction.