SparseInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation

Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, Yiran ChenICML 2026 · OpenReview

Introduction

Large Language Models (LLMs) with billions of parameters have transformed AI applications, but they require immense computational and memory resources during inference. While adaptive sparse activation—activating only a small subset of neurons for each token—offers a promising path to accelerate inference, existing methods rely on token-level predictions that limit efficiency gains.

Enter SparseInfer, a novel, "MLP-free" framework that accelerates LLM inference by identifying a fixed set of "core neurons" for an entire sentence. By leveraging the semantic stability of sentences, SparseInfer achieves up to a 10.33× speedup on GPUs and significantly reduces memory overhead without degrading model performance.

The Bottleneck in LLM Inference

During inference, LLMs process data in two stages: pre-filling (processing the input prompt) and decoding (generating text one token at a time). Moving data and computing across billions of parameters for every single token is incredibly slow and memory-intensive.

Previous "dynamic activation sparsity" methods (such as PowerInfer or DejaVu) attempt to solve this by using small auxiliary neural networks (MLPs) to predict which neurons will activate for the next token. However, this token-level approach introduces two major flaws:

  1. Computational Overhead: The predictor MLPs themselves can consume up to 10% of the total computation time.
  2. Efficiency Bottlenecks: Constantly changing the "activation map" (the set of active neurons) for every single token creates irregular memory access patterns, preventing hardware from reaching its full speed.

The SparseInfer Solution

SparseInfer shifts the paradigm from token-level to sentence-level prediction. The authors introduce the concept of Core Neurons—a stable set of essential neurons required to process a coherent sentence. Because SparseInfer is MLP-free, it eliminates predictor overhead by using the natural activation patterns observed during the initial prompt processing.

SparseInfer operates in two primary modes depending on the input scenario:

  • Stability-guided Prediction: For long, fluent inputs (e.g., document summarization), core neurons are identified during the pre-filling stage. Because the sentence's semantics are already established, this set of neurons is "frozen" for the entire decoding process, eliminating the need for per-token predictions.
  • Similarity-guided Prediction: For short or "unstable" inputs (e.g., a brief question), the model uses semantic clustering. It identifies the input's topic and selects a pre-determined cluster of neurons specialized for that semantic space (e.g., "History" vs. "Science").

Key Insights: Thinking in Sentences

  • Semantic Stability: As a sentence grows longer, the required neurons stabilize. The authors discovered that after 100–200 tokens, the core group of neurons barely changes during text generation.
  • Semantic Clustering: Core neurons are semantically informative. Sentences sharing a topic tend to activate the same neural clusters across different model layers.
  • Task-Specific Sparsity: Not all tasks require the same computational power. Simple tasks like Question Answering require only about 20% of the model's neurons to maintain full accuracy, whereas complex tasks like Translation require closer to 40%.
  • Universal Applicability: While many sparsity studies focus on ReLU activation functions, SparseInfer is effective on SiLU-based models like Llama 3, indicating that core neurons are a universal property of LLMs.

Performance and Results

Evaluated across a wide array of benchmarks—including C4, SQuAD, TruthfulQA, LongBench, and wmt16—SparseInfer demonstrated remarkable efficiency:

  • Massive Speedups: On an NVIDIA Titan Xp, SparseInfer achieved a 10.33× speedup over the standard HuggingFace implementation for the OPT-6.7B model. It also outperformed existing sparse methods like PowerInfer by roughly 2.7× to 3×.
  • Memory Efficiency: For OPT-6.7B, the memory footprint was reduced from 12.1 GB to just 7.28 GB. This allows larger models to fit entirely on a single GPU, bypassing slow data transfers between the CPU and GPU.
  • Lossless Performance: Across most tasks, the performance drop was negligible. In specific scenarios, such as the MultiFieldQA task in LongBench, SparseInfer actually outperformed the dense model, likely because sparse activation acts as a form of semantic noise filtering.

Limitations and Future Directions

While SparseInfer presents a massive leap in inference efficiency, it does have limitations. If an input abruptly changes topics mid-sentence (extreme semantic shifts), the fixed core neurons might fail. Future iterations may include a monitoring component to recalculate core neurons upon detecting major semantic shifts.

Additionally, while neurons clearly specialize in topics, the underlying reasons for this functional partitioning during training remain an open question for future research. Finally, the static nature of SparseInfer's activation maps makes it highly suitable for mobile devices and specialized AI accelerators (ASICs), marking a promising avenue for edge AI deployment.

Summary updated

Abstract

Large Language Models (LLMs) with billions of parameters have transformed AI applications but require immense computational and memory resources during inference. Adaptive sparse activation inference, which activates only a small number of neurons for each token, offers a novel way to accelerate model inference without degrading performance, showing great potential for resource-constrained hardware devices. Yet existing token-level MLP-based methods frequently alter activation maps, reducing efficiency gains. In this paper, we introduce \textbf{SparseInfer}, an MLP-free adaptive sparse activation inference method based on sentence-level prediction. We first propose the concept of core neurons and empirically demonstrate that, for an input sentence, LLMs only need the core neurons to maintain performance. Remarkably, we discovered that core neurons exhibit both stability and similarity in relation to the sentence's semantics—an insight overlooked by previous studies. Building on this finding, we design two semantic-based methods for predicting core neurons to fit different input scenarios, which enables core neurons to be determined during the pre-filling stage and fixed during the encoding stage. Our experiments verify SparseInfer exhibits good performance in various tasks and achieve 10.33$\times$ speed up.
BibTeX
@inproceedings{wang2026sparseinfer,
  author = {Qinsi Wang and Saeed Vahidian and Hancheng Ye and Jianyang Gu and Jianyi Zhang and Yiran Chen},
  title = {{SparseInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=1sKFoIvmYg},
}
Figure