SIRIUS : Contexual Sparisty with Correction for Efficient LLMs

Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Xi Victoria Lin, Beidi Chen

TL;DR

SIRIUS is a novel framework that rescues the performance of efficient, "contextually sparse" Large Language Models (LLMs) on complex reasoning tasks. By pairing a fast sparse model with a full-sized model that periodically verifies and corrects output chunks, SIRIUS achieves up to 35% latency reduction while maintaining near-original model accuracy.

The Problem: Sparsity Breaks Reasoning

As LLMs grow, the cost of inference becomes a major bottleneck. Contextual Sparsity (CS) is a popular training-free technique that speeds up models by activating only a small subset of neurons relevant to a specific prompt. While CS works beautifully for simple tasks like summarization or chat, this paper reveals a critical failure mode: it breaks high-level reasoning.

The authors show that even a massive 70B model, when sparsified, can perform worse on math problems than a much smaller 8B dense model. The logic is sound, but the sparse model makes tiny, fatal "token-level" mistakes—like a single wrong digit in a long calculation—that propagate and ruin the final answer.

Contextual sparse models struggle with reasoning tasks like GSM8K compared to simpler tasks, but correcting a small fraction of tokens recovers performance.

Methods: The SIRIUS Framework

To address this fragility, the authors introduce SIRIUS, named after the binary star system where a bright star (the full model) and a dim star (the sparse model) orbit each other.

  1. The "Two-Body" System: A fast, sparse model generates a "chunk" of tokens (e.g., 16 tokens).
  2. Parallel Verification: A full-sized model then takes that entire chunk and verifies it in a single parallel forward pass. Because LLM inference is memory-bound (waiting for weights to load from VRAM), verifying 16 tokens in parallel takes almost the same time as generating one token.
  3. Correction & Rollback: If the full model finds a token with a very low likelihood, the system "rolls back" to that point, and the full model provides the correct token to put the generation back on track.
  4. KV Cache Rewriting: Even if no error is found, the full model rewrites the Key-Value (KV) cache with its high-quality representations, which helps the sparse model stay accurate for the next chunk.
Overview of the SIRIUS correction mechanism, showing parallel verification and KV cache rewriting.

Key Insights

  • Reasoning is Fragile: Unlike summarization, where a few "off" words don't change the gist, reasoning requires perfect token-level precision. A single miscalculation in an intermediate step leads to a completely wrong result.
  • Sparse Models Lack Self-Awareness: You cannot rely on a sparse model's own confidence (entropy) to know when it has made a mistake. The authors found that sparse models are often "confidently wrong," meaning they don't show spikes in entropy when they fail.
  • The 11% Rule: A key discovery was that correcting just 11% of tokens is often enough to recover 100% of the original model's reasoning performance.
  • Memory-Bound Advantage: SIRIUS exploits the fact that modern GPUs are often idle while waiting for data. Using that "free" compute to run a full-model verification pass in parallel is the secret to its efficiency.

Key Results

SIRIUS was evaluated across 6 models (Llama-2 and Llama-3 families) using 8 diverse generation tasks, including arithmetic reasoning (GSM8K), coding (HumanEval), and general understanding.

  • Accuracy Recovery: On the GSM8K math benchmark, SIRIUS boosted a Llama-3-8B sparse model from 38% accuracy to 72% (very close to the full model's 75%), while only increasing the "effective" parameter count by a tiny margin.
  • Wallclock Speedup:
  • On-chip (8B model): Achieved roughly a 20% reduction in latency compared to the full dense model.
  • Offloading (70B model): Achieved a 35% speedup by keeping the sparse weights on the GPU and only loading the full weights when needed for correction.
  • Consistency: The method showed consistent gains across different model architectures and task types, proving it is a robust general-purpose correction layer.

Limitations and Future Work

  • Rollback Overhead: SIRIUS still relies on rolling back generation when an error is found. If a sparse model is extremely poor, frequent rollbacks could eventually negate the speed gains.
  • Hyperparameter Tuning: The "period" (how often the full model checks) and the "threshold" (how picky the full model is) are currently fixed hyperparameters that might need tuning for different tasks.
  • Distribution Matching: Perfectly matching the "strong" model's output distribution without sacrificing all efficiency remains an open research challenge.

Summary updated

Abstract

With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS, an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly a 20% reduction in latency for 8B model on-chip and a 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.
BibTeX
@inproceedings{zhou2024sirius,
  author = {Yang Zhou and Zhuoming Chen and Zhaozhuo Xu and Xi Victoria Lin and Beidi Chen},
  title = {{SIRIUS : Contexual Sparisty with Correction for Efficient LLMs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=5bR2l1b2eh},
}
Figure