Abstract
Large Language Models (LLMs) are computationally intensive, requiring full-layer passes for every generated token. This paper introduces LITE (Losses from the InTermediate layErs), an instruction-tuning strategy that enables intermediate layers to produce high-quality text. By implementing a dynamic, confidence-based early exiting mechanism at the token level, the proposed method achieves inference cost reductions of up to 46% for LLaMA-2 models while maintaining generation quality comparable to standard final-layer decoding.
Introduction
State-of-the-art LLMs, such as LLaMA-2, typically consist of dozens of transformer layers (e.g., 32 for the 7B variant, 40 for the 13B). During standard inference, every token must traverse the entire depth of the network. This linear dependency on depth makes deployment slow and expensive for real-time applications. While "early exiting" has been explored for classification tasks, applying it to open-ended text generation is challenging because intermediate layers in standard models typically produce incoherent outputs. This research addresses the challenge of making middle layers "generation-ready" without compromising the performance of the final layer.
Methodology
The researchers propose two primary innovations to facilitate efficient inference:
1. Instruction Tuning with LITE
During the fine-tuning phase using the Alpaca dataset, the model is trained using a weighted average of losses calculated from several intermediate layers (e.g., layers 8, 12, 16, etc.) in addition to the final layer. A critical aspect of this approach is the use of a shared language modeling head for all layers. This ensures that no additional parameters are introduced to the model architecture, maintaining its original footprint while aligning internal representations earlier in the network.
2. Dynamic Confidence-Based Early Exiting
During inference, the model generates text token-by-token. For each token, the system monitors the "confidence"—defined as the softmax probability of the top predicted token—at specific intermediate checkpoint layers. If this confidence exceeds a pre-defined threshold, the model "exits" early, skipping the remaining layers for that specific token. If the confidence is insufficient, computation continues to the next checkpoint or the final layer.
Experimental Results
The LITE-tuned models were evaluated across four human-instruction datasets: Vicuna, Koala, WizardLM, and Self-Instruct. Quality was assessed using Claude as an LLM-as-a-judge, comparing LITE-tuned early-exit outputs against standard final-layer outputs.
As shown in the figure above, the LITE strategy consistently reduces computational costs (measured in TFLOPS) across all models and datasets with minimal impact on output quality. Key findings include:
- Efficiency Gains: The method achieved average inference cost reductions of 37.86% for 7B models and 46.35% for 13B models.
- Quality Preservation: Claude-based evaluations indicated that the quality of responses from dynamic early exiting was nearly indistinguishable from standard generation.
- Semantic Consistency: Outputs from the early-exit model maintained high semantic similarity (scores ~0.90) to full-model outputs, indicating that the logic of the model remains intact even when short-circuited.
Discussion and Limitations
The LITE approach demonstrates that intermediate layers can be trained to align with final-layer logic, providing a reliable signal for early exiting. However, the authors note a trade-off regarding Key-Value (KV) caching. Standard KV caching is difficult to implement with dynamic exiting because hidden states for skipped layers are not computed, which may offset some speed gains in specific hardware configurations. Additionally, performance is sensitive to the selection of confidence thresholds, which may require tuning depending on the specific application's requirements for speed versus precision.
Conclusion
LITE provides a simple yet effective framework for accelerating LLM inference without architectural changes. By forcing intermediate layers to "think" like the final layer during training, models can dynamically bypass redundant computation during inference. Future work may explore using these generation-ready intermediate layers for speculative sampling or hallucination detection.
Summary updated