FPGA Co-Design for Efficient N:M Sparse and Quantized Model Inference

Fen-Yu Hsieh, Yun-Chang Teng, Ding-Yong Hong, Jan-Jan WuarXiv 2025 · DOI · arXiv

TL;DR

This paper presents a hardware-software co-design framework for accelerating large language model (LLM) inference on FPGAs. By combining N:M structured pruning and low-bit quantization, they achieve significant reductions in model size and computational cost, enabling more efficient deployment on resource-constrained hardware.

What Problem Does This Paper Address?

Large language models (LLMs) have revolutionized natural language processing, but their massive size and computational demands pose a significant barrier to deployment, especially in environments with limited resources. This challenge is exacerbated by the need for real-time inference and low energy consumption. While techniques like pruning (removing redundant weights) and quantization (reducing numerical precision) can alleviate these issues, they often fall short individually. Pruning alone doesn't reduce the computational cost of operations on remaining weights, and quantization doesn't reduce the number of operations. Combining them offers a more comprehensive solution, but this often requires specialized hardware support that is not universally available or flexible enough to accommodate various sparsity patterns. This paper addresses the gap between flexible algorithmic approaches to model compression and the rigid hardware constraints of current accelerators.

Methods

The core of this work is a hardware-software co-design methodology that creates an automated framework for LLM compression and an FPGA-based accelerator.

Key components of the framework:

  • Unified Pipeline: This pipeline consists of two stages:
  • Offline Preparation: This stage applies N:M structured pruning and 4-bit integer (INT4) quantization to dense FP16 weights. The N:M pruning is a "fine-grained structured sparsity" where a fixed number of non-zero weights (N) are kept within a larger group (M) of weights. This creates a regular sparsity pattern that is easier to exploit in hardware. The weights are then quantized to INT4, significantly reducing storage. Crucially, the pruned weights are stored in a dense layout with zeros in place, avoiding the need for explicit sparsity metadata during runtime.
  • Online Benchmark Execution: This stage involves loading the compressed weights and scaling factors onto the target device, dequantizing the INT4 weights back to FP16 while preserving sparsity, and then performing matrix multiplication with FP16 activations. This process is designed to be platform-agnostic, allowing for fair benchmarking across CPUs, NVIDIA GPUs (with and without sparse tensor cores), and their custom FPGA accelerator.
  • FPGA Accelerator: A custom systolic-array-based FPGA accelerator is designed to efficiently process the N:M sparse and INT4 quantized data. It features a dynamic zero-skipping mechanism within each processing element (PE). When a PE encounters a zero weight, it bypasses the multiplication and simply forwards the activation and weight, effectively reducing computations by approximately 50% for a 2:4 sparsity pattern. The accelerator is designed to be reconfigurable to accommodate diverse N:M patterns and bit-widths.
  • General Analytical Model: The authors developed a model to guide the design space exploration for optimal N:M patterns and bit-widths, aiming to maximize hardware utilization and throughput while minimizing accuracy loss.

Datasets

The paper does not mention specific datasets used for training LLMs. Instead, it focuses on evaluating the compression and acceleration techniques on synthetic 4096x4096 weight matrices and uses the LLaMA-7B model for predicted performance analysis. The evaluation is performed on these matrices and the linear layers of LLaMA-7B.

Key Insights and Ideas

  • Synergy of Pruning and Quantization: The paper strongly emphasizes that combining N:M structured pruning and low-bit quantization is crucial for achieving significant efficiency gains in LLM inference, far beyond what either technique can achieve alone.
  • Structured Sparsity for Hardware Efficiency: The N:M structured sparsity, particularly the 2:4 pattern, is shown to be highly amenable to hardware acceleration. The regular pattern and the ability to implement zero-skipping logic in the FPGA PEs are key to exploiting this sparsity effectively.
  • Dense Storage of Sparse Weights: Storing pruned weights in a dense format with zeros in place simplifies the hardware implementation by eliminating the need for complex sparsity metadata management during runtime. This is a clever way to bridge the gap between algorithmic sparsity and hardware execution.
  • FPGA as a Flexible Platform: The proposed FPGA accelerator offers a flexible architectural path for supporting a broader range of sparsity patterns beyond the fixed 2:4 constraints found in some GPU hardware, opening doors for future research into more aggressive sparsity schemes.
  • Platform-Agnostic Framework: The unified framework allows for fair benchmarking and demonstrates that the compression techniques are effective across different hardware platforms, highlighting the general applicability of their approach.

Key Results

  • Weight Size Reduction: Achieved up to a 4x reduction in effective weight footprint for 4096x4096 matrices with 2:4 pruning and INT4 quantization (when weights are bit-packed).
  • GEMM Speedup: Demonstrated a 1.71x speedup in matrix multiplication (GEMM) for the 2:4 sparse kernel compared to dense GPU baselines on 4096x4096 matrices.
  • End-to-End Compute Gain: Achieved a 1.29x end-to-end compute gain over dense GPU baselines.
  • LLaMA-7B Throughput Improvement: Scaling analysis on LLaMA-7B linear layers predicted a 1.36x improvement in per-token throughput for the 2:4 sparse GPU path compared to dense execution.
  • FPGA Performance: The FPGA accelerator achieved a throughput of 81968.13 GFLOP/s for GEMM on 4096x4096 matrices with a batch size of 512, outperforming dense GPU and CPU significantly.

Limitations and Future Work

Limitations:

  • Bit-Packing for INT4: While the paper mentions bit-packing for INT4 weights to achieve the 4x reduction, their prototype implementation stores quantized weights as an 8-bit integer tensor, which results in a 2x reduction. The full 4x reduction is achieved when weights are bit-packed.
  • LLaMA-7B Prediction: The LLaMA-7B results are predictions based on linear layers only and do not include other crucial components like attention mechanisms, softmax, or residual connections. Therefore, the actual end-to-end latency for LLaMA-7B would be higher.
  • Setup Time: The paper notes that setup time (I/O and GPU conversion) can dominate per-iteration costs, especially for small batch sizes, and is excluded from steady-state comparisons.

Future Work:

  • Full Empirical Evaluation: Conduct a complete evaluation of the FPGA implementation, including power and energy efficiency.
  • End-to-End LLaMA Inference: Extend the framework to support end-to-end inference of LLMs like LLaMA.
  • Richer Sparsity and Quantization Schemes: Explore higher-ratio N:M patterns (e.g., 4:16) and mixed-precision activations to further reduce costs.
  • Edge and Datacenter Deployment: Focus on enabling large-model deployment on edge devices and in datacenters.

Summary updated

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of language processing tasks. However, this success comes at the cost of substantial computation and memory requirements, which significantly impedes their deployment in resource-constrained environments. To address this challenge, this work introduces an automation framework that leverages weight pruning and low-bit quantization, and presents a hardware-software co-design method that generates accelerators on the Field-Programmable Gate Array (FPGA) platform. In particular, we implement a unified pipeline that applies N:M structured pruning and 4-bit integer quantization to reduce the memory footprint, followed by optimized dequantization and matrix multiplication to enhance LLM inference on several hardware platforms, including CPUs, NVIDIA GPUs with Dense and 2:4 Sparse Tensor Cores, and a custom systolic-array-based FPGA accelerator. Utilizing 2:4 sparsity combined with quantization on $4096 \times 4096$ matrices, our approach achieves a reduction of up to $4\times$ in weight storage and a $1.71\times$ speedup in matrix multiplication, yielding a $1.29\times$ end-to-end latency reduction compared to dense GPU baselines. Scaling analysis on the LLaMA-7B model further shows that structured sparsity enhances the throughput per token by $1.36\times$. These results demonstrate the synergy of fine-grained N:M sparsity and quantization for enabling efficient and deployable LLM inference, while the proposed FPGA accelerator offers a flexible architectural path for supporting a broader class of sparsity patterns beyond the fixed 2:4 hardware constraints.
BibTeX
@misc{hsieh2025fpga,
  author = {Fen-Yu Hsieh and Yun-Chang Teng and Ding-Yong Hong and Jan-Jan Wu},
  title = {{FPGA Co-Design for Efficient N:M Sparse and Quantized Model Inference}},
  year = {2025},
  eprint = {2512.24713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.24713},
}
Figure