Learning Low Precision Deep Neural Networks through Regularization.

Yoojin Choi, Mostafa El‐Khamy, Jungwon LeearXiv 2018 · arXiv

Abstract

We consider the quantization of deep neural networks (DNNs) to produce low-precision models for efficient inference of fixed-point operations. Compared to previous approaches to training quantized DNNs directly under the constraints of low-precision weights and activations, we learn the quantization of DNNs with minimal quantization loss through regularization. In particular, we introduce the learnable regularization coefficient to find accurate low-precision models efficiently in training. In our experiments, the proposed scheme yields the state-of-the-art low-precision models of AlexNet and ResNet-18, which have better accuracy than their previously available low-precision models. We also examine our quantization method to produce low-precision DNNs for image super resolution. We observe only 0.5 dB peak signal-to-noise ratio (PSNR) loss when using binary weights and 8-bit activations. The proposed scheme can be used to train low-precision models from scratch or to fine-tune a well-trained high-precision model to converge to a low-precision model. Finally, we discuss how a similar regularization method can be adopted in DNN weight pruning and compression, and show that 401x compression is achieved for LeNet-5.

Introduction

Modern Deep Neural Networks (DNNs) are notoriously over-parameterized, often containing millions of 32-bit floating-point parameters. While this complexity enables high accuracy, it presents significant challenges for deployment on resource-constrained hardware like mobile phones and IoT devices, which lack high-end GPUs and have limited battery life.

To address this, researchers employ quantization (reducing bit-width) and pruning (removing unnecessary weights). However, training these models is difficult due to the "gradient mismatch" problem: the forward pass uses low-precision weights to calculate loss, while the backward pass requires high-precision gradients to update the model. This mismatch often leads to suboptimal training and significant accuracy degradation. This paper introduces a novel training pipeline that uses learnable regularization coefficients to bridge the gap between high-precision training and low-precision execution.

Methodology

The core innovation is the Learnable Regularization Coefficient (λ). Instead of treating regularization strength as a fixed hyperparameter, it is treated as a parameter the network learns during training. This allows the model to dynamically balance task loss (accuracy) and quantization constraints.

The Compression Pipeline

The process follows a structured three-step sequence to transform a high-precision model into a compressed, low-precision version:

  1. Weight Pruning (Partial L2 Regularization): L2 regularization is applied only to the smallest weights (e.g., the r-th percentile). By making the regularization strength learnable, the network drives these small weights toward zero smoothly.
  2. Weight Quantization (MSQE Regularization): To force weights toward specific discrete "bins" (like -1, 0, 1), the authors use Mean Squared Quantization Error (MSQE). The learnable λ reinforces the convergence of high-precision "shadow" weights to their nearest quantized values.
  3. Activation Quantization: The system optimizes the scaling factors for activations to minimize quantization error during the forward pass.
  4. Entropy Coding: Finally, lossless compression (like Huffman coding) is applied to the quantized weights.
The proposed training pipeline for model compression, moving from high-precision to compressed models.

Key Insights: Visualizing Convergence

A critical observation in this research is how the weights evolve during the training process. By making λ learnable, the model "decides" when to prioritize accuracy and when to prioritize the quantization constraint. Early in training, λ is small, allowing weights to move freely. As the model matures, λ increases, pushing weights into their final quantized "slots."

This transition is clearly visible in weight histograms. As training progresses, the initial Gaussian-like distribution of weights gradually "clumps" into perfectly spaced delta functions, representing the target fixed-point values. This hardware-friendly uniform quantization is critical because uniform fixed-point arithmetic is significantly faster to implement on standard mobile hardware using simple bit-shifts.

Evolution of weight histograms during training, showing the convergence to discrete quantization levels.

Experimental Results

The proposed method was evaluated across multiple domains, demonstrating state-of-the-art performance in extreme compression scenarios:

  • Image Classification (ImageNet): MobileNet v1 achieved a 7.13x compression ratio using 5-bit weights and 8-bit activations with only a 0.6% drop in Top-1 accuracy. ShuffleNet achieved a 6.79x ratio with similar negligible loss. ResNet-18 outperformed existing methods like DoReFa-Net, particularly at very low bit-widths (e.g., 2-bit weights).
  • Image Super-Resolution: The method produced 8-bit models with only a 0.04 dB PSNR loss. Even using binary weights (1-bit) resulted in only a 0.5 dB loss, which is visually almost imperceptible.
  • Extreme Compression: For simpler architectures like LeNet-5 on MNIST, the method achieved a staggering 401x compression ratio.

Limitations and Future Work

While highly effective, the current pipeline requires separate stages for pruning and quantization. Future research could explore a unified, single-shot approach. Additionally, while the paper targets fixed-point arithmetic for theoretical efficiency, future benchmarks on specific mobile hardware (DSPs or NPUs) would provide more concrete power and speed metrics. Finally, extending this learnable regularization to Transformer architectures (like ViT or BERT) represents a valuable next step given their current dominance in the field.

Summary updated

Abstract

We consider the quantization of deep neural networks (DNNs) to produce low-precision models for efficient inference of fixed-point operations. Compared to previous approaches to training quantized DNNs directly under the constraints of low-precision weights and activations, we learn the quantization of DNNs with minimal quantization loss through regularization. In particular, we introduce the learnable regularization coefficient to find accurate low-precision models efficiently in training. In our experiments, the proposed scheme yields the state-of-the-art low-precision models of AlexNet and ResNet-18, which have better accuracy than their previously available low-precision models. We also examine our quantization method to produce low-precision DNNs for image super resolution. We observe only $0.5$~dB peak signal-to-noise ratio (PSNR) loss when using binary weights and 8-bit activations. The proposed scheme can be used to train low-precision models from scratch or to fine-tune a well-trained high-precision model to converge to a low-precision model. Finally, we discuss how a similar regularization method can be adopted in DNN weight pruning and compression, and show that $401\times$ compression is achieved for LeNet-5.
BibTeX
@misc{choi2018learning,
  author = {Yoojin Choi and Mostafa El‐Khamy and Jungwon Lee},
  title = {{Learning Low Precision Deep Neural Networks through Regularization.}},
  year = {2018},
  eprint = {1809.00095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.00095},
}
Figure