Accelerating LLM Inference by Enabling Intermediate Layer Decoding

Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, Chitta BaralarXiv 2023 · DOI · arXiv

Abstract

Large Language Models (LLMs) are computationally intensive, requiring full-layer passes for every generated token. This paper introduces LITE (Losses from the InTermediate layErs), an instruction-tuning strategy that enables intermediate layers to produce high-quality text. By implementing a dynamic, confidence-based early exiting mechanism at the token level, the proposed method achieves inference cost reductions of up to 46% for LLaMA-2 models while maintaining generation quality comparable to standard final-layer decoding.

Introduction

State-of-the-art LLMs, such as LLaMA-2, typically consist of dozens of transformer layers (e.g., 32 for the 7B variant, 40 for the 13B). During standard inference, every token must traverse the entire depth of the network. This linear dependency on depth makes deployment slow and expensive for real-time applications. While "early exiting" has been explored for classification tasks, applying it to open-ended text generation is challenging because intermediate layers in standard models typically produce incoherent outputs. This research addresses the challenge of making middle layers "generation-ready" without compromising the performance of the final layer.

Methodology

The researchers propose two primary innovations to facilitate efficient inference:

1. Instruction Tuning with LITE

During the fine-tuning phase using the Alpaca dataset, the model is trained using a weighted average of losses calculated from several intermediate layers (e.g., layers 8, 12, 16, etc.) in addition to the final layer. A critical aspect of this approach is the use of a shared language modeling head for all layers. This ensures that no additional parameters are introduced to the model architecture, maintaining its original footprint while aligning internal representations earlier in the network.

2. Dynamic Confidence-Based Early Exiting

During inference, the model generates text token-by-token. For each token, the system monitors the "confidence"—defined as the softmax probability of the top predicted token—at specific intermediate checkpoint layers. If this confidence exceeds a pre-defined threshold, the model "exits" early, skipping the remaining layers for that specific token. If the confidence is insufficient, computation continues to the next checkpoint or the final layer.

Experimental Results

The LITE-tuned models were evaluated across four human-instruction datasets: Vicuna, Koala, WizardLM, and Self-Instruct. Quality was assessed using Claude as an LLM-as-a-judge, comparing LITE-tuned early-exit outputs against standard final-layer outputs.

Comparison of Quality and Computational Cost (TFLOPS) across 7B and 13B models using standard final layer decoding versus confidence-based early exiting.

As shown in the figure above, the LITE strategy consistently reduces computational costs (measured in TFLOPS) across all models and datasets with minimal impact on output quality. Key findings include:

  • Efficiency Gains: The method achieved average inference cost reductions of 37.86% for 7B models and 46.35% for 13B models.
  • Quality Preservation: Claude-based evaluations indicated that the quality of responses from dynamic early exiting was nearly indistinguishable from standard generation.
  • Semantic Consistency: Outputs from the early-exit model maintained high semantic similarity (scores ~0.90) to full-model outputs, indicating that the logic of the model remains intact even when short-circuited.

Discussion and Limitations

The LITE approach demonstrates that intermediate layers can be trained to align with final-layer logic, providing a reliable signal for early exiting. However, the authors note a trade-off regarding Key-Value (KV) caching. Standard KV caching is difficult to implement with dynamic exiting because hidden states for skipped layers are not computed, which may offset some speed gains in specific hardware configurations. Additionally, performance is sensitive to the selection of confidence thresholds, which may require tuning depending on the specific application's requirements for speed versus precision.

Conclusion

LITE provides a simple yet effective framework for accelerating LLM inference without architectural changes. By forcing intermediate layers to "think" like the final layer during training, models can dynamically bypass redundant computation during inference. Future work may explore using these generation-ready intermediate layers for speculative sampling or hallucination detection.

Summary updated

Abstract

Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks; however, their large size makes their inference slow and computationally expensive. Focusing on this problem, we propose to instruction tune LLMs with additional explicit losses from the intermediate layers (LITE) and show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer. We perform 'dynamic confidence-based early exiting' at token level from the intermediate layers which improves the efficiency of text generation without compromising the quality of the generation. We conduct comprehensive experiments by instruction tuning LLaMA-2 models on the Alpaca dataset and holistically evaluate on four different human-instruction test sets. We show that dynamic early exiting achieves consistent and considerable inference computation cost improvements (37.86% for 7B and 46.35% for 13B model) while maintaining the generation quality of the responses. We further conduct a thorough analysis of the results over several important aspects, such as comparing the semantic similarity of the outputs and dissecting the efficiency improvements by comparing the number of tokens generated in the output. In summary, our work contributes to improving the efficiency of LLM inference while maintaining the generation quality, a crucial step en route to enabling their widespread adoption.
BibTeX
@misc{varshney2023accelerating,
  author = {Neeraj Varshney and Agneet Chatterjee and Mihir Parmar and Chitta Baral},
  title = {{Accelerating LLM Inference by Enabling Intermediate Layer Decoding}},
  year = {2023},
  eprint = {2310.18581},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18581},
}
Figure