LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning

Beomseok Kang, Jiwon Song, Jae-Joon Kim

Introduction

Small Language Models (SLMs) are increasingly deployed for complex tasks through multi-stage reasoning—a process where models decompose problems into sequential steps such as fact recall, analysis, and summarization. While this approach enhances accuracy, it introduces significant computational latency. Traditional acceleration techniques like layer skipping, which bypass specific transformer layers to save time, face unique challenges in multi-stage contexts.

Impact of layer skipping on accuracy and latency.

Two primary obstacles hinder effective layer skipping in reasoning tasks:

  1. Non-uniform Sensitivity: Different reasoning stages exhibit varying robustness. For instance, a "Recall" stage might tolerate aggressive skipping, whereas a "Summary" stage may suffer significant accuracy collapse with even minimal layer removal.
  2. The Latency Paradox: Counter-intuitively, skipping layers can sometimes increase total inference time. As models lose depth, they often become less confident, leading to the generation of redundant, low-quality tokens. Even if per-token latency decreases, the total time grows due to excessive verbosity.

The LiteStage Framework

LiteStage is a training-free framework designed to navigate these trade-offs by combining offline profiling with dynamic online adjustments.

Overview of the LiteStage framework.

1. Offline Configuration

The framework begins with an offline search phase to establish "layer budgets" for each reasoning stage:

  • Importance Estimation: LiteStage calculates the cosine similarity between a layer's input and output. Layers with high similarity are deemed redundant and prioritized for skipping.
  • Stage-wise Greedy Search: The system profiles the model on a validation set, iteratively determining the optimal number of layers to skip. It prioritizes the most time-consuming (longest) stages first, ensuring that optimization efforts are concentrated where they yield the highest latency gains while respecting a global accuracy threshold.

2. Online Adjustment

To mitigate the latency paradox, LiteStage employs a Confidence-based Early Exit strategy. During inference, the system monitors the average confidence (logits) of the most recent n tokens. If this confidence falls below a specific threshold, the system forces an "End of Sequence" (EOS) token. This prunes the verbose, low-quality tails of responses that typically emerge when layers are skipped, effectively recovering latency gains.

Experimental Evaluation

The researchers evaluated LiteStage using TinyLlama-1.1B and Qwen2.5-0.5B models across several reasoning benchmarks, including OpenBookQA (OBQA), CommonsenseQA (CSQA), and StrategyQA.

Key findings include:

  • Superior Efficiency: On the OBQA dataset, LiteStage achieved a 1.49x speedup with TinyLlama while maintaining ~60% accuracy. In contrast, existing baselines like AdaSkip and SkipDecode experienced total accuracy collapse at similar speedup levels.
  • Latency Recovery: The online early-exit mechanism proved essential. By terminating low-confidence generation, LiteStage successfully converted potential slowdowns caused by layer skipping into net speedups.
  • Generalizability: The framework demonstrated consistent performance across different model architectures, confirming its broad applicability to modern SLMs.

Conclusion and Limitations

LiteStage provides a practical, training-free solution for accelerating multi-stage reasoning. By recognizing the asymmetric trade-offs between different reasoning steps and addressing the latency paradox through confidence monitoring, it achieves significant speedups without sacrificing performance.

However, limitations remain. The method requires an initial offline profiling step, and its effectiveness diminishes in "Deep Reasoning" tasks (e.g., complex mathematics) where errors from skipped layers accumulate over long horizons. Future research may explore adaptive confidence thresholds and periodic full-decoding to address these challenges.

Summary updated

Abstract

Multi-stage reasoning has emerged as an effective strategy for enhancing the reasoning capability of small language models by decomposing complex problems into sequential sub-stages. However, this comes at the cost of increased latency. We observe that existing adaptive acceleration techniques, such as layer skipping, struggle to balance efficiency and accuracy in this setting due to two key challenges: (1) stage-wise variation in skip sensitivity, and (2) the generation of redundant output tokens. To address these, we propose LiteStage, a latency-aware layer skipping framework for multi-stage reasoning. LiteStage combines a stage-wise offline search that allocates optimal layer budgets with an online confidence-based generation early exit to suppress unnecessary decoding. Experiments on three benchmarks, e.g., OBQA, CSQA, and StrategyQA, show that LiteStage outperforms prior training-free layer skipping methods.
BibTeX
@misc{kang2025litestage,
  author = {Beomseok Kang and Jiwon Song and Jae-Joon Kim},
  title = {{LiteStage: Latency-aware Layer Skipping for Multi-stage Reasoning}},
  year = {2025},
  eprint = {2510.14211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14211},
}
Figure