Pruning Ternary Quantization

Dan Liu, Xi Chen, Jie Fu, Xue Liu

TL;DR

This paper introduces Pruning Ternary Quantization (PTQ), a novel method that simultaneously optimizes model size, accuracy, and inference speed by integrating L2 normalization, pruning, and weight decay. PTQ achieves highly compressed ternary models with minimal accuracy loss, outperforming existing compression techniques.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful but computationally expensive, making them difficult to deploy on resource-constrained devices. Model compression techniques like quantization (reducing the number of bits per weight) and pruning (removing less important weights) aim to address this. However, existing methods often optimize for one or two factors (speed, size, accuracy) at the expense of others. For instance, low-bit quantization speeds up inference but can significantly degrade accuracy, while pruning can maintain accuracy but doesn't always translate to faster inference without specialized hardware. The challenge is to find a method that effectively balances all three aspects for truly efficient and accurate models.

Methods

The core of PTQ lies in a clever combination of techniques:

  • Ternary Quantization: Weights are quantized to one of three values: -1, 0, or +1. This drastically reduces model size and computational overhead.
  • L2 Normalization: Applied to weights during training, this helps to make weight magnitudes more uniform and closer to discrete values.
  • Pruning: Unimportant weights are removed, further reducing model size and sparsity. The paper specifically uses unstructured pruning.
  • Weight Decay: This regularization technique, when minimized in conjunction with L2 normalization and pruning, is shown to push weight magnitudes towards being near-identical and discrete.
  • Straight-Through Estimator (STE): Used to approximate gradients during the non-differentiable quantization process, allowing for end-to-end training.
  • Resetting Trick: A novel step where weights are reset to their ternary values {-1, 0, 1} after pruning. This helps reduce the discrepancy between forward and backward passes in STE, leading to more accurate quantization.

The paper mathematically demonstrates how minimizing the weight decay term, under L2 normalization and pruning, leads to weights that are close to discrete and identical in magnitude, which is crucial for effective ternary quantization.

Datasets

The proposed method was evaluated on the following datasets:

  • ImageNet: Used for image classification tasks.
  • MS COCO: Used for object detection and segmentation tasks.

Key Insights and Ideas

  • Synergy of Compression Techniques: The paper demonstrates that combining L2 normalization, pruning, and weight decay is more effective than using them in isolation for achieving highly compressed and accurate ternary models.
  • Weight Decay as a Driver for Discreteness: The theoretical analysis shows that minimizing the weight decay term, under specific conditions, naturally pushes weight magnitudes towards discrete values, which is fundamental for ternary quantization.
  • Pruning Bridges the Gap: Pruning is shown to resolve the conflict between gradient updates and weight decay, enabling the L2-normalized weights to approach the desired discrete ternary values.
  • Resetting Trick for STE Accuracy: Resetting weights to ternary values after pruning significantly improves the accuracy of the STE-based quantization process by reducing gradient discrepancies.

Key Results

PTQ achieves state-of-the-art compression ratios with minimal accuracy degradation across various network architectures and tasks:

  • Image Classification (ImageNet):
    • ResNet-18: Compressed from 45 MB to 939 KB (49x compression) with only a 4% accuracy drop.
  • ResNet-50: Compressed from 99 MB to 3.1 MB (32x compression) with a 1.45% accuracy drop.
  • MobileNetV2: Achieved the best compression-accuracy trade-off at over 15x compression.
  • Object Detection (MS COCO):
    • Mask R-CNN: Compressed from 170 MB to 5 MB (34x compression) with a 2.8% average precision drop.
  • Activation Quantization: When combined with 2-bit activation quantization, PTQ achieved similar top-1 accuracy with approximately half the model size compared to other leading 2-bit methods.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Activation Quantization: While PTQ is compatible with activation quantization methods like LSQ, further exploration is needed to maximize inference efficiency by fully leveraging activation quantization.
  • Hardware Compatibility: The current method's 5-bit codebook representation might pose challenges for efficient storage on existing hardware architectures. More experiments across different devices are needed.
  • Training Time: Applying L2 normalization before pruning and quantization triples the training time.
  • Pruning Ratio Estimation: Determining the optimal pruning ratio (e.g., 70% for classification, 80% for detection) is currently based on empirical experience and requires multiple experiments. Developing methods to estimate this ratio more efficiently is important.

Future work includes combining PTQ with ternary activation quantization to further accelerate inference and exploring more efficient ways to estimate pruning ratios.

Summary updated

Abstract

Inference time, model size, and accuracy are three key factors in deep model compression. Most of the existing work addresses these three key factors separately as it is difficult to optimize them all at the same time. For example, low-bit quantization aims at obtaining a faster model; weight sharing quantization aims at improving compression ratio and accuracy; and mixed-precision quantization aims at balancing accuracy and inference time. To simultaneously optimize bit-width, model size, and accuracy, we propose pruning ternary quantization (PTQ): a simple, effective, symmetric ternary quantization method. We integrate L2 normalization, pruning, and the weight decay term to reduce the weight discrepancy in the gradient estimator during quantization, thus producing highly compressed ternary weights. Our method brings the highest test accuracy and the highest compression ratio. For example, it produces a 939kb (49$\times$) 2bit ternary ResNet-18 model with only 4\% accuracy drop on the ImageNet dataset. It compresses 170MB Mask R-CNN to 5MB (34$\times$) with only 2.8\% average precision drop. Our method is verified on image classification, object detection/segmentation tasks with different network structures such as ResNet-18, ResNet-50, and MobileNetV2.
BibTeX
@misc{liu2021pruning,
  author = {Dan Liu and Xi Chen and Jie Fu and Xue Liu},
  title = {{Pruning Ternary Quantization}},
  year = {2021},
  eprint = {2107.10998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.10998},
}
Figure