Position-based Scaled Gradient for Model Quantization and Pruning

Jangho Kim, KiYoon Yoo, Nojun Kwak

TL;DR

The paper introduces Position-based Scaled Gradient (PSG), a novel regularization technique that scales gradients based on a weight's distance from target compression points. By altering the optimization landscape, PSG steers neural networks toward solutions that maintain high accuracy in both full-precision and highly compressed (quantized or pruned) forms, enabling flexible "on-the-fly" deployment.

What Problem Does This Paper Address?

Deploying deep neural networks on resource-constrained edge devices typically requires model compression techniques like quantization (reducing weight bit-width) or pruning (removing unimportant weights).

Existing approaches fall into two main camps, both with significant drawbacks:

  1. Quantization-Aware Training (QAT): Fine-tunes a model specifically for low-precision hardware. While accurate, it destroys the model's full-precision performance and requires full datasets and high computational overhead for retraining.
  2. Post-Training Quantization (PTQ): Quantizes a pre-trained full-precision model without retraining. This is fast but suffers severe accuracy drops below 6-bit precision because standard Stochastic Gradient Descent (SGD) produces a continuous, bell-shaped weight distribution that maps poorly to discrete quantization grids.

The authors address this gap by asking: Can we train a single model from scratch that performs exceptionally well in full-precision mode, yet can be instantly compressed for edge deployment without any retraining or data access?

Methods

The core idea is Position-based Scaled Gradient Descent (PSGD). Instead of modifying the loss function with an explicit regularization term, PSGD scales the gradient element-wise based solely on the current position of the weight vector.

The scaling function is designed as: $$s(x) = |x - \bar{x}(x)| + \epsilon$$ where $\bar{x}$ is the closest target grid point (or zero for pruning) and $\epsilon$ is a tiny constant to prevent gradients from vanishing.

  • The Intuition: If a weight is far from a target grid point, it receives a larger gradient step, allowing it to escape that position quickly. If it is close to a target point, the gradient shrinks, encouraging the weight to settle precisely on the grid point.
  • Warped Space Theory: The authors theoretically prove that applying PSGD in the original weight space is mathematically equivalent to performing standard gradient descent in a warped weight space. This warping expands the regions near target points and contracts regions far away, naturally biasing the optimization toward compression-friendly local minima.
Figure 2: Toy example of warping a loss function L(x) = cos ((x −3.07)2). Left denotes the original loss function. Right is drawn by warping the orignal function by Eq. (5) with the target ¯x = 0.

Datasets

The method was evaluated on standard image classification benchmarks:

  • MNIST: Used for toy examples and loss surface visualization.
  • CIFAR-10 & CIFAR-100: Used for both pruning and low-bit quantization experiments.
  • ImageNet (ILSVRC 2015): Used to validate scalability on large-scale real-world problems.

Key Insights and Ideas

  • Versatile Deployment: Unlike traditional methods that force a choice between a full-precision or a quantized model, PSGD yields a single weight distribution that satisfies both. A system can dynamically switch between full-precision and low-bit modes depending on available hardware resources.
  • Sharp Valleys are Better for Compression: Standard SGD tends to find wide, flat valleys in the loss landscape. PSGD intentionally steers the model into sharper valleys that align perfectly with discrete quantization coordinates. While harder to find with standard optimizers, these solutions are highly robust to quantization noise.
  • Bridges the Distribution Gap: Standard training results in a massive discrepancy between full-precision and quantized weight distributions. PSGD forces the full-precision weights to naturally cluster around the quantization grid points, minimizing the mean squared error (MSE) when the bits are stripped away.
Figure 1: Results of ResNet-34 on CIFAR-100. (a) Mean-squared quantization error (line) and classi- fication error (bar) across different bits. Blue: SGD, Red: PSGD. (b) Example of weight distribution (Conv2_1 layer [19]) trained with standard SGD and our PSGD. For PSGD, the distribution of the full precision weights closely resembles the low precision distribution, yet maintains its accuracy.

Key Results

Model Pruning

When evaluated on CIFAR-100 using ResNet-32, PSGD maintained high accuracy even at extreme sparsity levels without any post-pruning fine-tuning:

  • At 90% sparsity (90% of weights set to zero), standard SGD collapsed to 1.00% accuracy.
  • State-of-the-art single-shot pruning methods like SNIP achieved 60.14%.
  • PSGD achieved 64.33% accuracy, outperforming the baselines significantly.

Model Quantization

On the challenging ImageNet dataset using ResNet-18 under a strict 4-bit weight and 4-bit activation (W4A4) constraint:

  • Standard SGD failed completely, dropping to 0.30% accuracy.
  • Previous gradient regularization techniques maxed out at 55.32%.
  • PSGD achieved 63.45% accuracy.
  • When combined with a post-training quantization method (LAPQ), PSGD-pretrained models reached 66.5% accuracy, proving it serves as an excellent initialization for other downstream compression techniques.

Limitations and Future Work

While PSGD performs remarkably well down to 3-bit quantization, extreme constraints (like 2-bit quantization) cause a noticeable drop in the initial full-precision accuracy due to the severity of the geometric constraints imposed during training. Additionally, the hyperparameter $\lambda_s$ (which scales the warping effect) requires manual tuning for different bit-widths and architectures.

Future work could explore automating hyperparameter selection or extending this position-based scaling concept to non-uniform quantization and alternative architectures like Transformers.

Summary updated

Abstract

We propose the position-based scaled gradient (PSG) that scales the gradient depending on the position of a weight vector to make it more compression-friendly. First, we theoretically show that applying PSG to the standard gradient descent (GD), which is called PSGD, is equivalent to the GD in the warped weight space, a space made by warping the original weight space via an appropriately designed invertible function. Second, we empirically show that PSG acting as a regularizer to a weight vector is favorable for model compression domains such as quantization and pruning. PSG reduces the gap between the weight distributions of a full-precision model and its compressed counterpart. This enables the versatile deployment of a model either as an uncompressed mode or as a compressed mode depending on the availability of resources. The experimental results on CIFAR-10/100 and ImageNet datasets show the effectiveness of the proposed PSG in both domains of pruning and quantization even for extremely low bits. The code is released in Github.
BibTeX
@inproceedings{kim2020position,
  author = {Jangho Kim and KiYoon Yoo and Nojun Kwak},
  title = {{Position-based Scaled Gradient for Model Quantization and Pruning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure