Abstract
Large Language Models (LLMs) face significant deployment challenges due to high computational costs. While layer pruning offers a static solution, it fails to account for token-level importance. We introduce OrthoRank, a training-free, dynamic token selection method that accelerates inference by skipping computations for tokens that have aligned geometrically with the model's attention sink. By prioritizing tokens orthogonal to the sink, OrthoRank maintains higher accuracy and lower perplexity than traditional pruning methods across various benchmarks and models.
Introduction
The computational demand of Large Language Models (LLMs) is a primary bottleneck for real-time and on-device applications. A common strategy to mitigate this is layer pruning, which involves removing entire layers to reduce the model's depth. However, this "one-size-fits-all" approach is suboptimal because it treats all tokens identically, regardless of their semantic complexity or processing state.
In practice, tokens within a sequence require varying levels of depth. Some tokens reach a stable representation early, while others require extensive processing through all layers. Existing dynamic inference techniques, such as "early exit" or "mixture-of-depth," often necessitate expensive retraining or the integration of additional "router" modules. OrthoRank addresses this by leveraging the internal geometry of the model to identify important tokens without any additional training or parameters.
Methodology
The Gravitation Phenomenon
The foundation of OrthoRank is the observation of a "gravitation phenomenon" within the hidden state space of LLMs. As tokens progress through deeper layers, their hidden states tend to align—measured by increasing cosine similarity—with the hidden state of the "attention sink" (typically the initial <bos> token).
This alignment suggests that as a token's representation becomes "finished" or stable, it moves toward the sink token's position in the feature space. Conversely, tokens that remain orthogonal (perpendicular) to the sink token are still undergoing significant transformation and likely contain unrefined, critical information.
The OrthoRank Algorithm
OrthoRank utilizes this geometric signal to perform dynamic token selection at each layer. The process is as follows:
- Similarity Calculation: At a specific layer $l$, the model computes the inner product between the hidden state of each token and the sink token.
- Selection: The algorithm selects the top $K$ tokens with the smallest inner products—those most orthogonal to the sink.
- Execution: Selected tokens proceed through the full Attention and Feed-Forward Network (FFN) blocks.
- Residual Path: Unselected tokens skip the layer's primary computation via the residual connection. Importantly, they still update their Key-Value (KV) cache entries to ensure that subsequent tokens can still attend to them accurately.
This "plug-and-play" approach allows for immediate application to pre-trained models like Llama and Mistral without fine-tuning.
Experimental Evaluation
The effectiveness of OrthoRank was evaluated across several model families (Llama-2, Llama-3, Mistral, Mixtral) using a comprehensive suite of benchmarks:
- Perplexity: WikiText-2 and C4 validation sets.
- Zero-Shot Reasoning: PIQA, WinoGrande, HellaSwag, and ARC.
- Long-Context Performance: LongBench (up to 8k tokens).
- Factuality: TruthfulQA.
Key Results
- Efficiency vs. Accuracy: At 10% and 20% sparsity levels, OrthoRank consistently outperformed state-of-the-art iterative layer pruning (SLEB) in both perplexity and zero-shot accuracy.
- Long-Context Robustness: OrthoRank demonstrated superior performance on long sequences. For Llama-3-8B at 20% sparsity, it achieved 25.23% accuracy on LongBench, significantly higher than SLEB’s 17.85%.
- Throughput: The method maintains a throughput increase nearly proportional to the sparsity level, as it avoids the overhead of complex routing modules.
- Factual Integrity: On TruthfulQA, OrthoRank preserved reasoning capabilities better than layer pruning, as evidenced by higher MC1 and BLEU scores.
Discussion and Limitations
A critical insight from this work is the necessity of the KV cache update for skipped tokens. Ablation studies revealed that failing to update KV pairs for unselected tokens severely degrades performance, as it disrupts the attention mechanism for the remaining active tokens.
However, OrthoRank has limitations. It depends on the emergence of an attention sink, which typically occurs after the first few layers; thus, it is less effective in the earliest stages of the model. Additionally, performance degrades when sparsity exceeds 40-50%, making it ideal for moderate speedups (1.1x to 1.3x).
Conclusion
OrthoRank provides a simple yet powerful training-free method for LLM acceleration. By exploiting the inherent geometric relationship between tokens and the attention sink, it enables efficient, dynamic inference that respects the varying computational needs of individual tokens. Future research may explore adaptive selection ratios to further optimize performance based on input complexity.
Summary updated