AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhenzhe Zheng, Fan Wu

Abstract

As Large Language Models (LLMs) scale to handle increasingly massive contexts, the computational and memory overhead of inference becomes a critical bottleneck. This paper introduces AdaSkip, an adaptive, sublayer-wise skipping strategy designed to accelerate both the prefilling and decoding phases of long-context LLM inference. By leveraging input-output (IO) similarity to identify redundant modules, AdaSkip dynamically bypasses less critical Attention or Feed-Forward Network (FFN) sublayers. Experimental results demonstrate that AdaSkip achieves up to a 1.51x speedup and a 17% improvement in acceleration over existing baselines while maintaining high generation quality across diverse long-context benchmarks.

1. Introduction

The evolution of LLMs toward processing contexts of up to 1 million tokens has introduced significant challenges in latency and memory management. While "layer skipping"—the practice of omitting entire transformer blocks—has emerged as a popular optimization, existing approaches suffer from three primary limitations: a lack of adaptivity to specific inputs, coarse granularity that ignores the distinct roles of Attention and FFN modules, and a focus on the decoding phase that neglects the substantial bottleneck of the prefilling phase (Time to First Token).

AdaSkip addresses these gaps by introducing a fine-grained, adaptive skipping mechanism that operates at the sublayer level and optimizes the entire inference pipeline.

2. Methodology

2.1 IO Similarity as an Importance Metric

The fundamental premise of AdaSkip is based on IO Similarity. The authors observe that if a sublayer’s output vector remains nearly identical to its input vector (indicated by high cosine similarity), the sublayer contributes minimally to the final representation and can be safely bypassed.

2.2 Sublayer-wise Granularity

Unlike traditional methods that skip monolithic transformer layers, AdaSkip treats Attention and FFN modules independently. This is particularly effective in long-context scenarios where Attention modules often exhibit higher redundancy and are the primary drivers of latency and KV cache growth. Prioritizing the skipping of Attention modules allows for greater speedups and memory savings without the performance degradation associated with skipping essential FFN components.

2.3 Phase-Specific Optimization

AdaSkip employs two distinct strategies to handle the inference lifecycle:

  • Offline Importance Learning (Prefilling): Since similarity cannot be measured before the prefilling phase begins, AdaSkip utilizes historical profiling data to predict which sublayers are generally unimportant for a given model architecture. A "Scale Factor" is applied to correct for minor deviations during execution.
  • Online Importance Learning (Decoding): During token generation, AdaSkip monitors an "online window" of the first few tokens to refine its skipping strategy dynamically, ensuring the model adapts to the specific context of the sequence as it grows.

3. Experimental Evaluation

3.1 Setup

The researchers evaluated AdaSkip across several models, including LLaMA3.1-8B-128k, InternLM-7B-8k, and Vicuna-v1.5-7B-16k. Benchmarks included:

  • Prefilling: MultiFieldQA, TriviaQA, and TREC (6,400 to 8,600 tokens).
  • Decoding: GovReport and MultiNews (summarization tasks with up to 512 generated tokens).

3.2 Key Results

  • Accuracy Retention: In high-compression scenarios (8 sublayers skipped), AdaSkip maintained 72.8% accuracy on the TREC benchmark using LLaMA3.1. In contrast, competitive baselines like SkipDecode and Unified Skipping saw performance collapse to near 0%.
  • Latency Reduction: AdaSkip achieved a physical speedup of up to 1.51x during the prefilling phase for InternLM. Overall, it provided a 17% acceleration improvement over other state-of-the-art skipping methods.
  • Generation Stability: On long-form summarization (GovReport), AdaSkip maintained a Rouge-L score of 30.5 even with 8 sublayers skipped, significantly outperforming the Early Exit baseline (15.3).

4. Discussion and Insights

4.1 The Redundancy of Attention

The study reveals that in long-context tasks, Attention sublayers often show more concentrated similarity scores than FFNs, making them ideal candidates for skipping. This provides a dual benefit: reducing computation and saving significant KV cache memory.

4.2 Model-Specific Importance

Importance patterns are not universal. For instance, InternLM’s redundant layers are concentrated in the middle of the architecture, while LLaMA’s are located toward the end. This finding underscores the necessity of AdaSkip’s adaptive approach over static, "one-size-fits-all" skipping patterns.

5. Conclusion and Future Work

AdaSkip provides a robust framework for accelerating long-context LLMs by identifying and skipping redundant sublayers across both prefilling and decoding phases. While the method is highly effective, its gains can be influenced by hardware-specific optimizations (e.g., sequence parallelism) and requires an initial profiling run for new architectures. Future research may explore even finer-grained skipping at the attention-head level to further optimize performance in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{he2025adaskip,
  author = {Zhuomin He and Yizhen Yao and Pengfei Zuo and Bin Gao and Qinya Li and Zhenzhe Zheng and Fan Wu},
  title = {{AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34579},
}
Figure