Abstract
Large Language Models (LLMs) typically employ a static inference process, passing every input through all available layers regardless of the task's complexity. This paper discusses AdaInfer, an adaptive inference framework designed to optimize computational efficiency by enabling "early exits." By monitoring internal confidence signals via a lightweight classifier, AdaInfer identifies instances where deeper layers are redundant. Experimental results across various benchmarks demonstrate that AdaInfer reduces computational costs by an average of 18%—and up to 43% for specific tasks—while maintaining accuracy within 1% of the full model.
Introduction
The deployment of Large Language Models (LLMs) like Llama2 and OPT is constrained by their immense computational requirements during inference. In standard architectures, every input sequence undergoes a fixed number of transformations across all layers. However, the cognitive demand of tasks varies significantly; a simple sentiment classification requires far less processing than a complex multi-step reasoning problem.
Existing efficiency methods, such as static model pruning, often permanently remove layers, which can degrade the model's generalizability and performance in in-context learning. To address this, researchers have proposed AdaInfer, an instance-aware adaptive inference algorithm. AdaInfer allows the model to dynamically determine the necessary depth for each input, bypassing unnecessary layers without altering the original model weights.
Methodology: The AdaInfer Framework
AdaInfer implements an "Early Exit" (EE) strategy on top of frozen, decoder-only LLMs. The framework consists of three primary components:
1. Feature Selection
At each transformer layer, AdaInfer extracts two statistical features from the output logits (raw prediction scores):
- Gap: The numerical difference between the probability of the most likely token and the second most likely token.
- Top Prob: The absolute probability of the most likely token.
These features serve as proxies for the model's confidence in its current prediction.
2. Lightweight Classification
The extracted features are processed by a low-overhead classifier, such as a Support Vector Machine (SVM) or a Conditional Random Field (CRF). The classifier is trained to predict whether the intermediate layer's output will match the final layer's output. If the classifier predicts a match with high confidence, the inference process terminates early.
3. Adaptive Termination
Because AdaInfer is "plug-and-play," it does not require retraining the underlying LLM. This makes it compatible with other acceleration techniques, such as quantization, and ensures that the model's fundamental capabilities remain intact.
Experimental Evaluation
The researchers evaluated AdaInfer using the Llama2 (7B, 13B, 70B) and OPT-13B models across a diverse set of benchmarks, including Question Answering (MMLU, CommonsenseQA, SQuAD) and Text Classification (SST-2, AG News).
Key Findings
- Universal Confidence Signals: The "Gap" and "Top Prob" features were found to be robust indicators of convergence. As inputs progress through deeper layers, these values typically increase and stabilize, signaling that the model has "finished thinking."
- Task-Dependent Depth: The study confirmed that task difficulty dictates the required layer depth. For instance, sentiment analysis tasks often reached stable conclusions by the middle layers, whereas reasoning-heavy tasks like MMLU required the full stack.
- Performance and Efficiency: AdaInfer achieved an average pruning ratio of 17.8%. In sentiment analysis (SST-2), the pruning ratio reached 43%. Crucially, the accuracy drop remained below 1% across all tested tasks.
- Wall-Clock Speedup: Real-world testing on NVIDIA V100 GPUs showed end-to-end runtime speedups ranging from 1.03x to 1.30x.
Comparative Analysis
AdaInfer was compared against ShortGPT, a state-of-the-art static pruning method. While static pruning often leads to significant accuracy degradation on complex tasks (like SQuAD), AdaInfer maintains performance by adaptively retaining depth for difficult instances. Furthermore, in some simple tasks, AdaInfer slightly outperformed the full dense model, suggesting that excessive depth can occasionally introduce noise or "over-thinking."
Limitations and Future Work
While AdaInfer demonstrates significant promise, it currently focuses on tasks with single-token or short-answer outputs. Future research is needed to extend this adaptive approach to long-form sequential generation, where the difficulty may fluctuate token-by-token. Additionally, exploring features within attention heads or hidden states may further refine the exit decision-making process.
Conclusion
AdaInfer provides a robust, instance-aware solution for LLM inference efficiency. By dynamically adjusting the number of active layers based on input complexity, it offers a significant reduction in computational overhead without sacrificing the model's accuracy or generalizability.
Summary updated