Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

Tai An, Ruwu Cai, Yanzhe Zhang, Yang Liu, Hao Chen, Pengcheng Xie, Sheng Chang, Yiwu Yao, Gongyi WangarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) continue to scale, the "prefill" stage—the initial phase where the model processes a prompt—has emerged as a significant computational bottleneck. This stage is characterized by high compute density, making it difficult to optimize without sacrificing model performance. While weight-based pruning has been extensively studied, it often leads to substantial accuracy degradation.

Amber Pruner introduces a novel, training-free approach to this problem by leveraging N:M activation sparsity. By selectively pruning input activations in linear layers and integrating this with W8A8 quantization (a framework termed Outstanding-sparse), researchers have achieved over 55% acceleration in linear computations with less than 1% accuracy loss across standard benchmarks.

Methodology

Amber Pruner Algorithm

The core of the Amber Pruner is a training-free algorithm that applies structured N:M sparsity (e.g., 2:4 or 8:16 ratios) to activations. Unlike simple magnitude pruning, Amber Pruner utilizes Robust-Norm Scoring. This method evaluates activation importance by analyzing the norm of corresponding weight channels, stabilized by removing weight outliers and normalizing mean/variance across layers.

Overview of the N:M activation sparsity process, where an activation matrix is masked and compressed before multiplication.

Furthermore, the system employs a Layer Skipping Strategy. Sensitivity analysis revealed that specific modules, such as o_proj and up_proj, are particularly fragile. Amber Pruner automatically identifies and skips these layers to maintain model quality while maximizing speedups elsewhere.

Outstanding-sparse Framework

To further enhance efficiency, Amber Pruner is combined with W8A8 quantization (SmoothQuant). While quantization and pruning often conflict, the Outstanding-sparse framework redefines scaling factors to expand the activation range. This synergy helps reveal structured sparsity patterns, allowing both techniques to work in tandem effectively.

Key Insights: Why Activations?

A fundamental observation driving this research is that activations in LLMs are naturally "whiter" than weights. Approximately 50% of activation values are typically near zero, whereas weight tensors tend to be more uniformly distributed. This makes activations inherently more suitable for pruning.

Comparison of activation and weight distributions in the gate_proj layer, highlighting the natural sparsity of activations.

Additionally, the researchers found a "Wanda reversal" effect: while the Wanda algorithm uses activations to prune weights, Amber Pruner uses weight statistics to identify critical activations. This approach proves highly compatible even with Mixture-of-Experts (MoE) models, providing fine-grained sparsity within already sparse active experts.

Evaluation and Results

The framework was tested on LLaMA3.1-8B, Qwen2-7B, and Qwen3-30B-A3B (MoE) across a variety of zero-shot and reasoning tasks (ARC, MMLU, GSM8K, LongBench).

  • Efficiency: Amber Pruner accelerates more than 55% of linear computations during the prefill stage.
  • Accuracy: At an 8:16 sparsity ratio, the models maintained competitive performance, incurring less than 1% average accuracy loss.
  • Reasoning Stability: On the GSM8K benchmark, the proposed method matched or slightly exceeded Bfloat16 baseline performance, indicating that multi-step reasoning capabilities remain intact despite significant compression.

Conclusion and Future Work

Amber Pruner demonstrates that activation sparsity is a viable, high-performance alternative to weight pruning for LLM acceleration. While current hardware support for Sparse-Dense Matrix Multiplication (SpMM) is still maturing compared to weight-based kernels, the theoretical gains are substantial. Future research will likely focus on software-hardware co-optimization to fully realize the latency benefits on modern NPUs and GPUs, as well as exploring the impact of this sparsity on KV cache management in long-context scenarios.

Summary updated

Abstract

In the era of large language models (LLMs), N:M sparsity has emerged as a structured compression technique critical for accelerating inference. While prior work has primarily focused on weight sparsity, it often suffers from significant accuracy degradation. Activation sparsity, though promising, is typically training-dependent and faces challenges in generalization. To address these limitations, we introduce Amber Pruner, a training-free N:M activation sparsity method designed specifically for the prefill stage, targeting the acceleration of linear projection layers in LLMs. Extensive experiments across multiple models and sparsity ratios (2:4, 4:8, and 8:16) demonstrate that Amber Pruner can effectively sparsify and accelerate more than 55% of linear computations without requiring model retraining. To further enhance generality and efficiency, we propose Outstanding-sparse, a unified framework that integrates Amber Pruner with post-training W8A8 quantization. Our approach preserves strong performance across a range of downstream tasks, with notable advantages in generative tasks. This work pioneers a new frontier in activation sparsity, providing foundational insights that are poised to guide the co-evolution of algorithms and architectures in the design of next-generation AI systems.
BibTeX
@misc{an2025amber,
  author = {Tai An and Ruwu Cai and Yanzhe Zhang and Yang Liu and Hao Chen and Pengcheng Xie and Sheng Chang and Yiwu Yao and Gongyi Wang},
  title = {{Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models}},
  year = {2025},
  eprint = {2508.02128},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.02128},
}
Figure