Introduction
As Large Language Models (LLMs) continue to scale, the "prefill" stage—the initial phase where the model processes a prompt—has emerged as a significant computational bottleneck. This stage is characterized by high compute density, making it difficult to optimize without sacrificing model performance. While weight-based pruning has been extensively studied, it often leads to substantial accuracy degradation.
Amber Pruner introduces a novel, training-free approach to this problem by leveraging N:M activation sparsity. By selectively pruning input activations in linear layers and integrating this with W8A8 quantization (a framework termed Outstanding-sparse), researchers have achieved over 55% acceleration in linear computations with less than 1% accuracy loss across standard benchmarks.
Methodology
Amber Pruner Algorithm
The core of the Amber Pruner is a training-free algorithm that applies structured N:M sparsity (e.g., 2:4 or 8:16 ratios) to activations. Unlike simple magnitude pruning, Amber Pruner utilizes Robust-Norm Scoring. This method evaluates activation importance by analyzing the norm of corresponding weight channels, stabilized by removing weight outliers and normalizing mean/variance across layers.
Furthermore, the system employs a Layer Skipping Strategy. Sensitivity analysis revealed that specific modules, such as o_proj and up_proj, are particularly fragile. Amber Pruner automatically identifies and skips these layers to maintain model quality while maximizing speedups elsewhere.
Outstanding-sparse Framework
To further enhance efficiency, Amber Pruner is combined with W8A8 quantization (SmoothQuant). While quantization and pruning often conflict, the Outstanding-sparse framework redefines scaling factors to expand the activation range. This synergy helps reveal structured sparsity patterns, allowing both techniques to work in tandem effectively.
Key Insights: Why Activations?
A fundamental observation driving this research is that activations in LLMs are naturally "whiter" than weights. Approximately 50% of activation values are typically near zero, whereas weight tensors tend to be more uniformly distributed. This makes activations inherently more suitable for pruning.
Additionally, the researchers found a "Wanda reversal" effect: while the Wanda algorithm uses activations to prune weights, Amber Pruner uses weight statistics to identify critical activations. This approach proves highly compatible even with Mixture-of-Experts (MoE) models, providing fine-grained sparsity within already sparse active experts.
Evaluation and Results
The framework was tested on LLaMA3.1-8B, Qwen2-7B, and Qwen3-30B-A3B (MoE) across a variety of zero-shot and reasoning tasks (ARC, MMLU, GSM8K, LongBench).
- Efficiency: Amber Pruner accelerates more than 55% of linear computations during the prefill stage.
- Accuracy: At an 8:16 sparsity ratio, the models maintained competitive performance, incurring less than 1% average accuracy loss.
- Reasoning Stability: On the GSM8K benchmark, the proposed method matched or slightly exceeded Bfloat16 baseline performance, indicating that multi-step reasoning capabilities remain intact despite significant compression.
Conclusion and Future Work
Amber Pruner demonstrates that activation sparsity is a viable, high-performance alternative to weight pruning for LLM acceleration. While current hardware support for Sparse-Dense Matrix Multiplication (SpMM) is still maturing compared to weight-based kernels, the theoretical gains are substantial. Future research will likely focus on software-hardware co-optimization to fully realize the latency benefits on modern NPUs and GPUs, as well as exploring the impact of this sparsity on KV cache management in long-context scenarios.
Summary updated