Introduction
Small Language Models (SLMs) are increasingly deployed for complex tasks through multi-stage reasoning—a process where models decompose problems into sequential steps such as fact recall, analysis, and summarization. While this approach enhances accuracy, it introduces significant computational latency. Traditional acceleration techniques like layer skipping, which bypass specific transformer layers to save time, face unique challenges in multi-stage contexts.
Two primary obstacles hinder effective layer skipping in reasoning tasks:
- Non-uniform Sensitivity: Different reasoning stages exhibit varying robustness. For instance, a "Recall" stage might tolerate aggressive skipping, whereas a "Summary" stage may suffer significant accuracy collapse with even minimal layer removal.
- The Latency Paradox: Counter-intuitively, skipping layers can sometimes increase total inference time. As models lose depth, they often become less confident, leading to the generation of redundant, low-quality tokens. Even if per-token latency decreases, the total time grows due to excessive verbosity.
The LiteStage Framework
LiteStage is a training-free framework designed to navigate these trade-offs by combining offline profiling with dynamic online adjustments.
1. Offline Configuration
The framework begins with an offline search phase to establish "layer budgets" for each reasoning stage:
- Importance Estimation: LiteStage calculates the cosine similarity between a layer's input and output. Layers with high similarity are deemed redundant and prioritized for skipping.
- Stage-wise Greedy Search: The system profiles the model on a validation set, iteratively determining the optimal number of layers to skip. It prioritizes the most time-consuming (longest) stages first, ensuring that optimization efforts are concentrated where they yield the highest latency gains while respecting a global accuracy threshold.
2. Online Adjustment
To mitigate the latency paradox, LiteStage employs a Confidence-based Early Exit strategy. During inference, the system monitors the average confidence (logits) of the most recent n tokens. If this confidence falls below a specific threshold, the system forces an "End of Sequence" (EOS) token. This prunes the verbose, low-quality tails of responses that typically emerge when layers are skipped, effectively recovering latency gains.
Experimental Evaluation
The researchers evaluated LiteStage using TinyLlama-1.1B and Qwen2.5-0.5B models across several reasoning benchmarks, including OpenBookQA (OBQA), CommonsenseQA (CSQA), and StrategyQA.
Key findings include:
- Superior Efficiency: On the OBQA dataset, LiteStage achieved a 1.49x speedup with TinyLlama while maintaining ~60% accuracy. In contrast, existing baselines like AdaSkip and SkipDecode experienced total accuracy collapse at similar speedup levels.
- Latency Recovery: The online early-exit mechanism proved essential. By terminating low-confidence generation, LiteStage successfully converted potential slowdowns caused by layer skipping into net speedups.
- Generalizability: The framework demonstrated consistent performance across different model architectures, confirming its broad applicability to modern SLMs.
Conclusion and Limitations
LiteStage provides a practical, training-free solution for accelerating multi-stage reasoning. By recognizing the asymmetric trade-offs between different reasoning steps and addressing the latency paradox through confidence monitoring, it achieves significant speedups without sacrificing performance.
However, limitations remain. The method requires an initial offline profiling step, and its effectiveness diminishes in "Deep Reasoning" tasks (e.g., complex mathematics) where errors from skipped layers accumulate over long horizons. Future research may explore adaptive confidence thresholds and periodic full-decoding to address these challenges.
Summary updated