Introduction
Quantizing Large Language Models (LLMs) is a critical step for enabling deployment on consumer-grade hardware. However, existing post-training quantization (PTQ) methods often force a trade-off between inference speed and model accuracy. Round-To-Nearest (RTN) methods, such as AWQ or GPTQ, offer high speed by mapping weights to uniform integer grids but suffer from high quantization error because LLM weights typically follow a Gaussian distribution. Conversely, codebook-based methods like AQLM provide high accuracy by fitting the distribution precisely but introduce significant latency due to non-sequential memory access patterns that bypass GPU caches.
SBVR (Summation of BitVector Representation) addresses this "speed vs. accuracy" dilemma. It represents weight vectors as a weighted sum of bitvectors, capturing the natural Gaussian distribution of weights while maintaining high-speed inference through bitwise operations.
Methodology: Summation of BitVector Representation
The fundamental innovation of SBVR is the decomposition of a weight vector into multiple bitvectors (binary elements) and associated floating-point coefficients. A weight is reconstructed by summing the coefficients corresponding to the active bits in its representation.
Hardware-Efficient Computation
By storing weights as bitvectors, the inner product—the primary operation in matrix-vector multiplication (GEMV)—can be computed using bitwise AND and population count (popcount) operations. This approach replaces traditional floating-point fused multiply-add (FMA) operations with high-throughput bitwise logic.
To maximize performance, the researchers developed a custom CUDA kernel that executes these bitwise operations directly on the compressed format, avoiding the overhead of decompressing weights into memory. This hardware-algorithm co-design reduces memory traffic and utilizes the high-throughput bit-manipulation units of the GPU.
Key Insights and Distribution Analysis
Standard uniform quantization (RTN) is often compared to measuring a curve with a straight ruler. SBVR effectively adapts the "ruler" to the Gaussian shape of the data. As illustrated in the distribution comparison, SBVR captures the central peak and the tails of the weight distribution significantly better than RTN, which tends to over-simplify the distribution into a flat histogram.
Furthermore, to mitigate the computational cost of the initial quantization search, SBVR employs a caching mechanism. Since many weight vectors exhibit similar optimal coefficient sets, caching allows the system to skip the search process for approximately 90% of vectors, making the PTQ process more practical.
Experimental Results
SBVR was evaluated across several modern LLM families, including Llama 3/3.1, Qwen3, and DeepSeek-R1-Distil-Qwen-7B, using benchmarks like WikiText-2 and a suite of commonsense reasoning tasks.
- Accuracy: In 4-bit quantization scenarios, SBVR consistently outperformed AWQ and GPTQ in terms of perplexity. Its accuracy was found to be competitive with state-of-the-art codebook methods like VPTQ.
- Inference Speed: SBVR achieved an end-to-end speedup of 2.21× to 3.04× compared to FP16 baselines on an NVIDIA RTX 3090.
- Kernel Performance: In direct comparisons on Llama 3.1-8B, the SBVR kernel (10.24ms) was faster than both the GPTQ kernel (10.51ms) and the specialized QQQ kernel (12.66ms).
Conclusion and Future Work
SBVR represents a significant step forward in LLM quantization by combining distribution-aware accuracy with hardware-native speed. While currently optimized for NVIDIA GPUs and focused on weight quantization, future work may explore more sophisticated activation quantization schemes and porting the specialized kernels to other hardware architectures like Apple Silicon or AMD GPUs.
Summary updated