SBVR: Summation of BitVector Representation for Efficient LLM Quantization

Wonjun Bang, Jongseok Park, Hongseung Yu, Kyungmin Bin, Kyunghan LeearXiv 2025 · DOI · arXiv

Introduction

Quantizing Large Language Models (LLMs) is a critical step for enabling deployment on consumer-grade hardware. However, existing post-training quantization (PTQ) methods often force a trade-off between inference speed and model accuracy. Round-To-Nearest (RTN) methods, such as AWQ or GPTQ, offer high speed by mapping weights to uniform integer grids but suffer from high quantization error because LLM weights typically follow a Gaussian distribution. Conversely, codebook-based methods like AQLM provide high accuracy by fitting the distribution precisely but introduce significant latency due to non-sequential memory access patterns that bypass GPU caches.

SBVR (Summation of BitVector Representation) addresses this "speed vs. accuracy" dilemma. It represents weight vectors as a weighted sum of bitvectors, capturing the natural Gaussian distribution of weights while maintaining high-speed inference through bitwise operations.

Methodology: Summation of BitVector Representation

The fundamental innovation of SBVR is the decomposition of a weight vector into multiple bitvectors (binary elements) and associated floating-point coefficients. A weight is reconstructed by summing the coefficients corresponding to the active bits in its representation.

Hardware-Efficient Computation

By storing weights as bitvectors, the inner product—the primary operation in matrix-vector multiplication (GEMV)—can be computed using bitwise AND and population count (popcount) operations. This approach replaces traditional floating-point fused multiply-add (FMA) operations with high-throughput bitwise logic.

The process of computing an inner product using bitwise AND and popcount operations.

To maximize performance, the researchers developed a custom CUDA kernel that executes these bitwise operations directly on the compressed format, avoiding the overhead of decompressing weights into memory. This hardware-algorithm co-design reduces memory traffic and utilizes the high-throughput bit-manipulation units of the GPU.

Key Insights and Distribution Analysis

Standard uniform quantization (RTN) is often compared to measuring a curve with a straight ruler. SBVR effectively adapts the "ruler" to the Gaussian shape of the data. As illustrated in the distribution comparison, SBVR captures the central peak and the tails of the weight distribution significantly better than RTN, which tends to over-simplify the distribution into a flat histogram.

Comparison showing how SBVR (green) fits the Gaussian weight distribution better than uniform RTN (grey).

Furthermore, to mitigate the computational cost of the initial quantization search, SBVR employs a caching mechanism. Since many weight vectors exhibit similar optimal coefficient sets, caching allows the system to skip the search process for approximately 90% of vectors, making the PTQ process more practical.

Experimental Results

SBVR was evaluated across several modern LLM families, including Llama 3/3.1, Qwen3, and DeepSeek-R1-Distil-Qwen-7B, using benchmarks like WikiText-2 and a suite of commonsense reasoning tasks.

  • Accuracy: In 4-bit quantization scenarios, SBVR consistently outperformed AWQ and GPTQ in terms of perplexity. Its accuracy was found to be competitive with state-of-the-art codebook methods like VPTQ.
  • Inference Speed: SBVR achieved an end-to-end speedup of 2.21× to 3.04× compared to FP16 baselines on an NVIDIA RTX 3090.
  • Kernel Performance: In direct comparisons on Llama 3.1-8B, the SBVR kernel (10.24ms) was faster than both the GPTQ kernel (10.51ms) and the specialized QQQ kernel (12.66ms).

Conclusion and Future Work

SBVR represents a significant step forward in LLM quantization by combining distribution-aware accuracy with hardware-native speed. While currently optimized for NVIDIA GPUs and focused on weight quantization, future work may explore more sophisticated activation quantization schemes and porting the specialized kernels to other hardware architectures like Apple Silicon or AMD GPUs.

Summary updated

Abstract

With the advent of large language models (LLMs), numerous Post-Training Quantization (PTQ) strategies have been proposed to alleviate deployment barriers created by their enormous parameter counts. Quantization achieves compression by limiting the number of representable points in the data. Therefore, the key to achieving efficient quantization is selecting the optimal combination of representation points, or codes, for the given data. Existing PTQ solutions adopt two major approaches to this problem: Round-To-Nearest (RTN)-based methods and codebook-based methods. RTN-based methods map LLM weights onto uniformly distributed integer grids, failing to account for the Gaussian-like weight distribution of LLM weights. Codebook-based methods mitigate this issue by constructing distribution-aware codebooks; however, they suffer from random and strided memory access patterns, resulting in degraded inference speed that is exacerbated by the limited size of GPU L1 cache. To overcome these limitations, we propose a novel LLM quantization method, SBVR (Summation of BitVector Representation), that enables Gaussian-like code representation in a hardware-friendly manner for fast inference. SBVR maps weight values to non-uniform representation points whose distribution follows the actual distribution of LLM weights, enabling more accurate compression. Additionally, we design a custom CUDA kernel that allows matrix-vector multiplication directly in the SBVR format without decompression, thereby enabling high-performance execution of SBVR-compressed models. Our evaluations of SBVR on various models demonstrate state-of-the-art perplexity and accuracy benchmark performance while delivering a 2.21x- 3.04x end-to-end token-generation speedup over naive FP16 models in the 4-bit quantization regime.
BibTeX
@misc{bang2025sbvr,
  author = {Wonjun Bang and Jongseok Park and Hongseung Yu and Kyungmin Bin and Kyunghan Lee},
  title = {{SBVR: Summation of BitVector Representation for Efficient LLM Quantization}},
  year = {2025},
  eprint = {2509.18172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.18172},
}
Figure