Abstract
Serving Large Language Models (LLMs) is computationally expensive and memory-intensive. While Early-Exit LLMs (EE-LLMs) offer latency improvements by skipping layers for simple tokens, they are often limited by the need to keep full models in memory and the inefficiency of single-model exit strategies. We introduce HELIOS, a novel framework that dynamically switches between multiple early-exit models and greedily loads only the most necessary layers. By reclaiming unused GPU memory to scale batch sizes, HELIOS achieves up to 1.48× higher throughput and a 15.14× increase in batch size compared to existing state-of-the-art serving frameworks.
Introduction
The primary challenges in LLM serving are high latency and limited throughput due to memory constraints. Early-Exit mechanisms allow "easy" tokens to stop processing at intermediate layers. However, current implementations suffer from two major flaws: the single-model bottleneck, where tokens failing to exit early must traverse all remaining layers, and the memory paradox, where all layers must be loaded "just in case," preventing batch size scaling. HELIOS addresses these by using an adaptive, multi-model approach that optimizes both latency and memory utilization.
Methodology
HELIOS implements a five-stage process to optimize inference:
- Candidate Selection: Identifies a pool of models (e.g., OPT-1.3B, OPT-6.7B) suitable for the workload.
- Real-Time Profiling: Builds a Performance History Table (PHT) by running initial requests to map exit probabilities.
- Greedy Loading: Loads only the "Low-Exit" layers most frequently used, freeing GPU memory for larger batches.
- Adaptive Switching & Loading: Monitors a Confidence Breach Counter (CBC). If confidence thresholds aren't met, the system either loads more layers or switches to a more capable model.
- Reassessment: Periodically re-evaluates model performance to adapt to changing request streams.
Key Insights
A critical observation in HELIOS is that early exits are often complementary across different models. A token requiring deep processing in a small model might exit early in a more "intelligent" larger model. Furthermore, the authors found that tokens often remain unchanged after an early exit layer even if they don't meet strict confidence thresholds, justifying "greedy" early exits to save resources without significant accuracy loss.
Evaluation and Results
HELIOS was evaluated across a wide range of benchmarks, including ShareGPT, CNN-Dailymail, and HumanEval. The results demonstrate significant improvements in efficiency and performance.
- Throughput and Batch Size: HELIOS improves throughput by 45% over vanilla decoding and supports batch sizes up to 15.14× larger due to memory re-purposing.
- Latency: The framework achieves a 2.23× reduction in Time To First Token (TTFT) and a 46.6% reduction in token generation latency (TPOT).
- Energy Efficiency: In constrained environments, HELIOS provides ~10% energy savings while maintaining high accuracy.
Conclusion and Future Work
HELIOS demonstrates that combining multi-model switching with greedy layer loading can transform latency optimizations into massive capacity gains. Future work includes integrating HELIOS with quantization and KV-cache compression to further enhance memory efficiency. While effective, the system requires careful management of the model pool and fine-tuning of reassessment intervals for volatile workloads.
Summary updated