Abstract
Sign-Splitting Vector Quantization (SSVQ) is a novel compression paradigm designed to address the inherent limitations of traditional Vector Quantization (VQ) during model fine-tuning. Standard VQ forces all weights within a cluster to update in the same direction, leading to "gradient dominance" where a minority of weights dictate the updates for the entire group. SSVQ decouples the sign bit from the magnitude codebook, making the signs learnable through a Straight-Through Estimator (STE). This flexibility allows for state-of-the-art accuracy at extreme compression ratios (up to 21×) and delivers a 3× hardware speedup on edge devices by significantly reducing DRAM traffic.
Introduction: The Fine-Tuning Bottleneck
Deep Neural Networks (DNNs) are increasingly deployed on edge devices with severe memory constraints. Vector Quantization (VQ) has been a go-to solution, grouping weights into clusters represented by a single "codeword." However, VQ suffers from a significant drawback during fine-tuning: because all weights in a cluster point to the same codeword, they are forced to update in unison.
The authors identify this as "gradient dominance." In standard VQ, the top 10% of gradients exhibit a 0.74 cosine similarity with the final update, while the bottom 50%—the "silent majority"—have almost no influence (0.19). This forces many weights to move in directions that increase their local error, severely limiting accuracy recovery post-quantization.
Methodology: Sign-Splitting and Iterative Freezing
SSVQ breaks the "one direction" constraint by treating signs and magnitudes as separate entities. The process involves three primary innovations:
- Sign-Magnitude Decoupling: Instead of clustering raw weights, SSVQ performs K-means clustering only on the absolute values of the weights (all-positive weights).
- Learnable Sign Masks: Sign bits are treated as continuous latent variables optimized via backpropagation using a Straight-Through Estimator (STE). This allows weights to "flip" their sign if the gradient indicates it would improve performance.
- Iterative Freezing: To maintain training stability, the frequency of sign changes is tracked via Exponential Moving Averages (EMA). Unstable signs are eventually frozen using a "majority-vote" strategy as training progresses.
Key Insights: Why Sign-Splitting Works
By removing the sign before clustering, the data distribution becomes simpler and more uniform. This allows SSVQ to utilize smaller codebooks or higher-dimensional vectors while maintaining lower reconstruction errors than standard VQ.
Crucially, this flexibility solves the gradient dominance problem. By allowing the "silent majority" of weights to evolve their signs independently, the model can navigate the loss landscape more effectively during fine-tuning. On the hardware side, while the 1-bit sign mask adds overhead, it enables larger vector dimensions ($d$), which reduces the total number of indices stored and cuts DRAM traffic.
Experimental Results
The researchers evaluated SSVQ across diverse tasks, demonstrating its robustness:
- Image Classification: On DeiT-tiny at a 21× compression ratio, SSVQ improved Top-1 accuracy by 12% absolute over standard VQ.
- Generative Models: For Stable Diffusion, SSVQ achieved a 30% lower FID score compared to conventional VQ under 2-bit quantization, effectively eliminating common artifacts like color bleeding and object disappearance.
- Large Language Models (LLMs): On Llama 3.2-1B, SSVQ reduced Wiki2 perplexity from 20.8 (VQ) to 11.9, nearing the full-precision baseline of 9.7.
- Hardware Efficiency: Cycle-accurate simulations showed a 3× speedup over 8-bit compressed models by reducing the frequency of slow external memory fetches.
Conclusion and Future Work
SSVQ represents a significant leap in neural network compression, particularly for extreme low-bit scenarios. While the 1-bit-per-weight sign mask introduces a minor storage floor, the gains in accuracy and hardware efficiency are substantial. Future research will focus on physical hardware implementations (FPGA/ASIC) and further tuning the iterative freezing hyperparameters for emerging architectures.
Summary updated