SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splitting

Shuaiting Li, Juncan Deng, Chenxuan Wang, Kedong Xu, Rongtao Deng, Hong Gu, Haibin Shen, Kejie Huang

Abstract

Sign-Splitting Vector Quantization (SSVQ) is a novel compression paradigm designed to address the inherent limitations of traditional Vector Quantization (VQ) during model fine-tuning. Standard VQ forces all weights within a cluster to update in the same direction, leading to "gradient dominance" where a minority of weights dictate the updates for the entire group. SSVQ decouples the sign bit from the magnitude codebook, making the signs learnable through a Straight-Through Estimator (STE). This flexibility allows for state-of-the-art accuracy at extreme compression ratios (up to 21×) and delivers a 3× hardware speedup on edge devices by significantly reducing DRAM traffic.

Introduction: The Fine-Tuning Bottleneck

Deep Neural Networks (DNNs) are increasingly deployed on edge devices with severe memory constraints. Vector Quantization (VQ) has been a go-to solution, grouping weights into clusters represented by a single "codeword." However, VQ suffers from a significant drawback during fine-tuning: because all weights in a cluster point to the same codeword, they are forced to update in unison.

The authors identify this as "gradient dominance." In standard VQ, the top 10% of gradients exhibit a 0.74 cosine similarity with the final update, while the bottom 50%—the "silent majority"—have almost no influence (0.19). This forces many weights to move in directions that increase their local error, severely limiting accuracy recovery post-quantization.

Methodology: Sign-Splitting and Iterative Freezing

SSVQ breaks the "one direction" constraint by treating signs and magnitudes as separate entities. The process involves three primary innovations:

  1. Sign-Magnitude Decoupling: Instead of clustering raw weights, SSVQ performs K-means clustering only on the absolute values of the weights (all-positive weights).
  2. Learnable Sign Masks: Sign bits are treated as continuous latent variables optimized via backpropagation using a Straight-Through Estimator (STE). This allows weights to "flip" their sign if the gradient indicates it would improve performance.
  3. Iterative Freezing: To maintain training stability, the frequency of sign changes is tracked via Exponential Moving Averages (EMA). Unstable signs are eventually frozen using a "majority-vote" strategy as training progresses.
The SSVQ pipeline: separating sign from magnitude for clustering and employing iterative freezing for sign stability.

Key Insights: Why Sign-Splitting Works

By removing the sign before clustering, the data distribution becomes simpler and more uniform. This allows SSVQ to utilize smaller codebooks or higher-dimensional vectors while maintaining lower reconstruction errors than standard VQ.

Comparison of VQ and SSVQ update flexibility. SSVQ allows weights to move in opposite directions despite sharing a codeword.

Crucially, this flexibility solves the gradient dominance problem. By allowing the "silent majority" of weights to evolve their signs independently, the model can navigate the loss landscape more effectively during fine-tuning. On the hardware side, while the 1-bit sign mask adds overhead, it enables larger vector dimensions ($d$), which reduces the total number of indices stored and cuts DRAM traffic.

Experimental Results

The researchers evaluated SSVQ across diverse tasks, demonstrating its robustness:

  • Image Classification: On DeiT-tiny at a 21× compression ratio, SSVQ improved Top-1 accuracy by 12% absolute over standard VQ.
  • Generative Models: For Stable Diffusion, SSVQ achieved a 30% lower FID score compared to conventional VQ under 2-bit quantization, effectively eliminating common artifacts like color bleeding and object disappearance.
  • Large Language Models (LLMs): On Llama 3.2-1B, SSVQ reduced Wiki2 perplexity from 20.8 (VQ) to 11.9, nearing the full-precision baseline of 9.7.
  • Hardware Efficiency: Cycle-accurate simulations showed a 3× speedup over 8-bit compressed models by reducing the frequency of slow external memory fetches.

Conclusion and Future Work

SSVQ represents a significant leap in neural network compression, particularly for extreme low-bit scenarios. While the 1-bit-per-weight sign mask introduces a minor storage floor, the gains in accuracy and hardware efficiency are substantial. Future research will focus on physical hardware implementations (FPGA/ASIC) and further tuning the iterative freezing hyperparameters for emerging architectures.

Summary updated

BibTeX
@inproceedings{li2025ssvq,
  author = {Shuaiting Li and Juncan Deng and Chenxuan Wang and Kedong Xu and Rongtao Deng and Hong Gu and Haibin Shen and Kejie Huang},
  title = {{SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splitting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure