QRazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

Dongyoung Lee, Seungkyu Choi, Ik Joon ChangOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

Abstract

Large-scale language models (LLMs) have demonstrated outstanding performance in language processing tasks, yet their deployment is often hindered by high memory demands and computational complexity. Although low-bit quantization techniques, such as 4-bit quantization, present a potential solution, they frequently lead to significant accuracy degradation or require substantial effort for such aggressive quantization approaches. To overcome these challenges, we introduce QRazor, a reliable and effortless quantization scheme designed to enable 4-bit quantization for weights, activations, and KV cache in transformer-based LLMs. The scheme involves two main stages: quantization and compression. During the quantization stage, weights, activations, and KV cache values are quantized with wider 8 or 16-bit integers as a basis to achieve nearly identical accuracy to the original full-precision LLM models, using the absolute max scaling. Subsequently, all data are compressed to 4-bit using our proposed significant data razoring (SDR) technique, which retains only the four most salient bits while discarding the others. Furthermore, we present an integer-based arithmetic unit dedicated to QRazor, enabling direct low-precision arithmetic operations without decompressing the SDR data. Despite the reduced quantization effort, QRazor achieves LLM accuracies better or comparable to state-of-the-art 4-bit methods. By also validating the hardware efficiency, our decompression-free arithmetic unit achieves 61.2\% and 57.8\% reduction in area and power consumption, respectively.
BibTeX
@misc{lee2025qrazor,
  author = {Dongyoung Lee and Seungkyu Choi and Ik Joon Chang},
  title = {{QRazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring}},
  year = {2025},
  url = {https://openreview.net/forum?id=lwcnZmyojm},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure