Sirius: Contextual Sparsity with Correction for Efficient LLMs

Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Victoria Lin, Beidi ChenarXiv 2024 · OpenReview

Abstract

With the blossom of large language models (LLMs), inference efficiency becomes increasingly important. Various approximation methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without quality degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, CS significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models often share general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces Sirius, an efficient correction mechanism, which significantly recovers CS models quality on reasoning tasks while maintaining its efficiency gain. Sirius is evaluated on 6 models with 8 difficult generation tasks in reasoning, math, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for Sirius and show that Sirius achieves roughly 20% reduction in latency for 8B model on-chip and 35% reduction for 70B model offloading.

The Reasoning Gap in Sparse LLMs

Large Language Models (LLMs) are notoriously resource-hungry. To make them faster and cheaper, researchers often turn to Contextual Sparsity (CS)—a technique that activates only a small subset of a model's neurons based on the specific input. While CS is appealing because it is "training-free"—allowing users to simply shut off 50% of neurons in a model like Llama-3 during inference—it has a significant drawback: the Reasoning Gap.

Researchers discovered that while a sparse Llama-3-8B can still summarize a news article effectively, its performance on math benchmarks like GSM8K collapses. The sparse model often follows the correct logic but makes small, fatal errors (like a simple miscalculation) that propagate and ruin the final answer.

Contextual sparsity maintains performance on summarization (CNN/DailyMail) but causes a collapse in reasoning accuracy (GSM8K) as density decreases.

The Sirius Framework

To address this, researchers from CMU, Stevens Institute of Technology, and Meta AI introduced Sirius, named after the binary star system consisting of a bright star and a dim companion. In this framework, the "dim" sparse model handles the bulk of the generation, while the "bright" full model steps in to verify and correct.

Key technical components include:

  • Parallel Verification: Instead of checking every token, Sirius lets the sparse model generate a "chunk" of tokens. The full model then verifies the entire chunk in parallel. Because LLM inference is memory-bound, verifying 64 tokens takes roughly the same time as verifying one.
  • KV Cache Rewriting: When the full model performs a check, it directly rewrites the Key-Value (KV) cache of the sparse model, ensuring the sparse model's future outputs are grounded in the full model's superior representations.
  • Hardware-Friendly Tree Building: To increase the chances of the full model accepting a chunk, the sparse model builds a small "tree" of likely token paths rather than just picking the single most likely token.
Overview of the Sirius mechanism: the full model periodically verifies token chunks and rewrites the sparse model's KV cache to maintain accuracy.

Key Insights and Results

The researchers evaluated Sirius across 6 models and 8 diverse tasks, including reasoning (GSM8K, AQuA-RAT), coding (HumanEval), and knowledge-based tasks (MMLU).

  • The 11% Rule: A critical finding was that correcting just 11% of tokens is often enough to recover 100% of the full model's performance. Most of a reasoning chain is "easy" for a sparse model; it only needs help at critical junctions.
  • Performance Recovery: On the GSM8K math benchmark, a sparse Llama-3-8B-Instruct model's accuracy dropped to 38%. Sirius boosted it back to 70.5%, nearing the full model's 75.4%.
  • Latency Speedup: Sirius achieved a 20% reduction in latency for 8B models running on-chip (Nvidia A100/H100) and a 35% reduction for 70B models using CPU-offloading.
  • Model Sensitivity: Interestingly, the more "well-trained" a model is (e.g., Llama-3 vs. Llama-2), the more its performance degrades under contextual sparsity, suggesting high-performance models utilize their parameters in a more dense, interconnected way.

Conclusion

Sirius demonstrates that we can enjoy the speed of sparse models without sacrificing the reasoning power of dense ones. While the current system relies on a fixed correction period, future work may explore adaptive signals to trigger corrections only when a mistake is detected. Additionally, "cross-model" Sirius—using a massive model to correct a much smaller one—could potentially create systems with the speed of a 10B model but the reasoning power of a 70B model.

Summary updated

Abstract

With the blossom of large language models (LLMs), inference efficiency becomes increasingly important. Various approximation methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without quality degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, CS significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models often share general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces Sirius, an efficient correction mechanism, which significantly recovers CS models quality on reasoning tasks while maintaining its efficiency gain. Sirius is evaluated on 6 models with 8 difficult generation tasks in reasoning, math, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for Sirius and show that Sirius achieves roughly 20% reduction in latency for 8B model on-chip and 35% reduction for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.
BibTeX
@misc{zhou2024sirius,
  author = {Yang Zhou and Zhuoming Chen and Zhaozhuo Xu and Victoria Lin and Beidi Chen},
  title = {{Sirius: Contextual Sparsity with Correction for Efficient LLMs}},
  year = {2024},
  eprint = {2409.03856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.03856},
}
Figure