HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving

Avinash Kumar, Shashank Nag, Jason Clemons, Lizy Kurian John, Poulami DasarXiv 2025 · DOI · arXiv

Abstract

Serving Large Language Models (LLMs) is computationally expensive and memory-intensive. While Early-Exit LLMs (EE-LLMs) offer latency improvements by skipping layers for simple tokens, they are often limited by the need to keep full models in memory and the inefficiency of single-model exit strategies. We introduce HELIOS, a novel framework that dynamically switches between multiple early-exit models and greedily loads only the most necessary layers. By reclaiming unused GPU memory to scale batch sizes, HELIOS achieves up to 1.48× higher throughput and a 15.14× increase in batch size compared to existing state-of-the-art serving frameworks.

Introduction

The primary challenges in LLM serving are high latency and limited throughput due to memory constraints. Early-Exit mechanisms allow "easy" tokens to stop processing at intermediate layers. However, current implementations suffer from two major flaws: the single-model bottleneck, where tokens failing to exit early must traverse all remaining layers, and the memory paradox, where all layers must be loaded "just in case," preventing batch size scaling. HELIOS addresses these by using an adaptive, multi-model approach that optimizes both latency and memory utilization.

Methodology

HELIOS implements a five-stage process to optimize inference:

  1. Candidate Selection: Identifies a pool of models (e.g., OPT-1.3B, OPT-6.7B) suitable for the workload.
  2. Real-Time Profiling: Builds a Performance History Table (PHT) by running initial requests to map exit probabilities.
  3. Greedy Loading: Loads only the "Low-Exit" layers most frequently used, freeing GPU memory for larger batches.
  4. Adaptive Switching & Loading: Monitors a Confidence Breach Counter (CBC). If confidence thresholds aren't met, the system either loads more layers or switches to a more capable model.
  5. Reassessment: Periodically re-evaluates model performance to adapt to changing request streams.
The HELIOS workflow: from candidate selection to adaptive token generation and model reassessment.

Key Insights

A critical observation in HELIOS is that early exits are often complementary across different models. A token requiring deep processing in a small model might exit early in a more "intelligent" larger model. Furthermore, the authors found that tokens often remain unchanged after an early exit layer even if they don't meet strict confidence thresholds, justifying "greedy" early exits to save resources without significant accuracy loss.

Evaluation and Results

HELIOS was evaluated across a wide range of benchmarks, including ShareGPT, CNN-Dailymail, and HumanEval. The results demonstrate significant improvements in efficiency and performance.

HELIOS significantly reduces the memory footprint of weights while simultaneously increasing throughput compared to vanilla and standard EE-LLM methods.
  • Throughput and Batch Size: HELIOS improves throughput by 45% over vanilla decoding and supports batch sizes up to 15.14× larger due to memory re-purposing.
  • Latency: The framework achieves a 2.23× reduction in Time To First Token (TTFT) and a 46.6% reduction in token generation latency (TPOT).
  • Energy Efficiency: In constrained environments, HELIOS provides ~10% energy savings while maintaining high accuracy.

Conclusion and Future Work

HELIOS demonstrates that combining multi-model switching with greedy layer loading can transform latency optimizations into massive capacity gains. Future work includes integrating HELIOS with quantization and KV-cache compression to further enhance memory efficiency. While effective, the system requires careful management of the model pool and fine-tuning of reassessment intervals for volatile workloads.

Summary updated

Abstract

Early-Exit Large Language Models (EE-LLMs) enable high throughput inference by allowing tokens to exit early at intermediate layers. However, their throughput is limited by the computational and memory savings. Existing EE-LLM frameworks rely on a single model and therefore, their token generation latencies are bottlenecked by tokens that do not exit early and traverse additional layers. Moreover, early exits are only known at runtime and depend on the request. Therefore, these frameworks load the weights of all model layers even though large portions remain unused when tokens exit early. The lack of memory savings limit us from scaling the batch sizes. We propose $\textit{HELIOS}$, a framework that improves both token generation latency and batch sizes to enable high-throughput in EE-LLMs. HELIOS exploits two insights. $\textit{First}$, early exits are often complimentary across models, tokens that do not exit early on one model often take an early-exit on another. HELIOS employs multiple models and dynamically switches between them to collectively maximize the number of tokens that exit early, and minimize token generation latencies. $\textit{Second}$, even when a predicted token does not exit early due to poor confidence, it often remains unchanged even after additional layer traversal. HELIOS greedily allows such tokens to exit early and only loads the weights of the most likely to be used layers, yielding memory savings which is then re-purposed to increase batch sizes. HELIOS employs real-time profiling to accurately identify the early-exit distributions, and adaptively switches between models by tracking tokens in real-time to minimize the performance degradation caused by greedy model loading and exiting. Our evaluations show that HELIOS achieves $1.48\times$ higher throughput and $15.14\times$ larger batch size compared to existing EE-LLM frameworks.
BibTeX
@misc{kumar2025helios,
  author = {Avinash Kumar and Shashank Nag and Jason Clemons and Lizy Kurian John and Poulami Das},
  title = {{HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving}},
  year = {2025},
  eprint = {2504.10724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10724},
}
Figure