Learning Sparse Low-Precision Neural Networks With Learnable Regularization

Yoojin Choi, Mostafa El‐Khamy, Jungwon LeeIEEE Access 2020 · DOI

TL;DR

This paper introduces a novel method for training deep neural networks (DNNs) with low-precision weights and activations, crucial for efficient deployment on resource-constrained devices. They achieve this by using a learnable regularization coefficient to minimize the mismatch between high-precision weights used during training and the low-precision weights used during inference, leading to state-of-the-art compression ratios with minimal accuracy loss.

What Problem Does This Paper Address?

Deploying deep neural networks (DNNs) on devices like smartphones or embedded systems is challenging due to their large size and computational demands. Traditional DNNs use high-precision floating-point numbers, which are power-hungry and memory-intensive. While quantizing networks to use low-precision fixed-point numbers (e.g., 8-bit or even fewer) significantly reduces these demands, it introduces a critical problem: the "gradient mismatch." During training, the forward pass uses quantized values to calculate the loss, but the backward pass (for gradient updates) often uses high-precision values. This discrepancy leads to suboptimal training and accuracy degradation. Existing methods try to bridge this gap, but often with complex approximations or fixed regularization parameters that don't adapt well.

Methods

The core of this paper's contribution lies in its regularization techniques:

  • Mean Squared Quantization Error (MSQE) Regularization: To reduce the mismatch between high-precision and quantized weights, the authors introduce MSQE regularization. This term penalizes the difference between a high-precision weight and its quantized counterpart.
  • Learnable Regularization Coefficient: Crucially, they make the coefficient (λ) of the MSQE regularizer learnable. This allows the regularization strength to adapt dynamically during training, promoting a smoother convergence to quantized values and better handling of the gradient mismatch. They also add a penalty term (-log λ) to encourage λ to increase.
  • Partial L2 Regularization for Pruning: For weight pruning (removing less important weights), they employ a partial L2 regularization. This encourages weights with small magnitudes to become exactly zero, allowing for safe pruning without significant accuracy loss.
  • Combined Pipeline: The paper proposes a pipeline that combines weight pruning, quantization (using MSQE regularization), and entropy coding for further compression.

Datasets

The paper evaluates its method on the following datasets:

  • ImageNet ILSVRC 2012: Used for classification tasks with models like AlexNet, ResNet-18, MobileNet, and ShuffleNet.
  • Open Images dataset: Used for image super-resolution tasks, pre-processed as described in their cited work.

Key Insights and Ideas

  • Learnable Regularization is Key: The most significant insight is that making the regularization coefficient learnable, rather than fixed, is crucial for effectively mitigating the gradient mismatch and achieving better accuracy in low-precision networks.
  • MSQE as a Bridge: MSQE regularization acts as an effective bridge, guiding high-precision weights towards their quantized targets and thus reducing the discrepancy between forward and backward passes.
  • Pruning and Quantization Synergy: Combining weight pruning with quantization, guided by these regularization techniques, leads to substantial compression ratios without sacrificing much accuracy.
  • Fixed-Point Inference is the Goal: The ultimate aim is to produce models that can run efficiently using fixed-point arithmetic on hardware, which is directly enabled by their low-precision weight and activation representations.

Key Results

  • State-of-the-Art Compression: For MobileNet and ShuffleNet on ImageNet, they achieve compression ratios of 7.13x and 6.79x, respectively, with minimal accuracy loss.
  • Image Super Resolution: For image super-resolution networks (CT-SRCNN), they achieve 8-bit low-precision models with negligible performance loss (e.g., only 0.04 dB PSNR loss for 8-bit weights and activations compared to 32-bit floating-point).
  • AlexNet and ResNet-18 Performance: They demonstrate competitive results on AlexNet and ResNet-18, showing that 4-bit quantization can achieve accuracy loss less than 1% on ImageNet.
  • Learnable vs. Fixed λ: Experiments show that their learnable regularization coefficient consistently outperforms fixed coefficients across various low-precision settings for ResNet-18.

Limitations and Future Work

  • Gradient Mismatch Still Exists: The authors acknowledge that their method still faces the gradient mismatch problem, albeit significantly reduced by their regularization approach.
  • Optimization Complexity: While their method is simpler than some prior relaxation techniques, optimizing the learnable regularization coefficient adds a new parameter to tune.
  • Future Work: The paper implicitly suggests exploring more aggressive quantization levels (e.g., 1-bit weights and activations) and applying their techniques to a wider range of network architectures and tasks. The effectiveness of their pipeline for both classification and regression tasks hints at broad applicability.

Summary updated

Abstract

We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training. Thus, the gradient descent becomes suboptimal, and accuracy loss follows. In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularization. In particular, we propose using a learnable regularization coefficient with the MSQE regularizer to reinforce the convergence of high-precision weights to their quantized values. We also investigate how partial L2 regularization can be employed for weight pruning in a similar manner. Finally, combining weight pruning, quantization, and entropy coding, we establish a low-precision DNN compression pipeline. In our experiments, the proposed method yields low-precision MobileNet and ShuffleNet models on ImageNet classification with the state-of-the-art compression ratios of 7.13 and 6.79, respectively. Moreover, we examine our method for image super resolution networks to produce 8-bit low-precision models at negligible performance loss.
BibTeX
@inproceedings{choi2020learning,
  author = {Yoojin Choi and Mostafa El‐Khamy and Jungwon Lee},
  title = {{Learning Sparse Low-Precision Neural Networks With Learnable Regularization}},
  booktitle = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2996936},
}
Figure