TL;DR
This paper introduces a novel method for training deep neural networks (DNNs) with low-precision weights and activations, crucial for efficient deployment on resource-constrained devices. They achieve this by using a learnable regularization coefficient to minimize the mismatch between high-precision weights used during training and the low-precision weights used during inference, leading to state-of-the-art compression ratios with minimal accuracy loss.
What Problem Does This Paper Address?
Deploying deep neural networks (DNNs) on devices like smartphones or embedded systems is challenging due to their large size and computational demands. Traditional DNNs use high-precision floating-point numbers, which are power-hungry and memory-intensive. While quantizing networks to use low-precision fixed-point numbers (e.g., 8-bit or even fewer) significantly reduces these demands, it introduces a critical problem: the "gradient mismatch." During training, the forward pass uses quantized values to calculate the loss, but the backward pass (for gradient updates) often uses high-precision values. This discrepancy leads to suboptimal training and accuracy degradation. Existing methods try to bridge this gap, but often with complex approximations or fixed regularization parameters that don't adapt well.
Methods
The core of this paper's contribution lies in its regularization techniques:
- Mean Squared Quantization Error (MSQE) Regularization: To reduce the mismatch between high-precision and quantized weights, the authors introduce MSQE regularization. This term penalizes the difference between a high-precision weight and its quantized counterpart.
- Learnable Regularization Coefficient: Crucially, they make the coefficient (λ) of the MSQE regularizer learnable. This allows the regularization strength to adapt dynamically during training, promoting a smoother convergence to quantized values and better handling of the gradient mismatch. They also add a penalty term (-log λ) to encourage λ to increase.
- Partial L2 Regularization for Pruning: For weight pruning (removing less important weights), they employ a partial L2 regularization. This encourages weights with small magnitudes to become exactly zero, allowing for safe pruning without significant accuracy loss.
- Combined Pipeline: The paper proposes a pipeline that combines weight pruning, quantization (using MSQE regularization), and entropy coding for further compression.
Datasets
The paper evaluates its method on the following datasets:
- ImageNet ILSVRC 2012: Used for classification tasks with models like AlexNet, ResNet-18, MobileNet, and ShuffleNet.
- Open Images dataset: Used for image super-resolution tasks, pre-processed as described in their cited work.
Key Insights and Ideas
- Learnable Regularization is Key: The most significant insight is that making the regularization coefficient learnable, rather than fixed, is crucial for effectively mitigating the gradient mismatch and achieving better accuracy in low-precision networks.
- MSQE as a Bridge: MSQE regularization acts as an effective bridge, guiding high-precision weights towards their quantized targets and thus reducing the discrepancy between forward and backward passes.
- Pruning and Quantization Synergy: Combining weight pruning with quantization, guided by these regularization techniques, leads to substantial compression ratios without sacrificing much accuracy.
- Fixed-Point Inference is the Goal: The ultimate aim is to produce models that can run efficiently using fixed-point arithmetic on hardware, which is directly enabled by their low-precision weight and activation representations.
Key Results
- State-of-the-Art Compression: For MobileNet and ShuffleNet on ImageNet, they achieve compression ratios of 7.13x and 6.79x, respectively, with minimal accuracy loss.
- Image Super Resolution: For image super-resolution networks (CT-SRCNN), they achieve 8-bit low-precision models with negligible performance loss (e.g., only 0.04 dB PSNR loss for 8-bit weights and activations compared to 32-bit floating-point).
- AlexNet and ResNet-18 Performance: They demonstrate competitive results on AlexNet and ResNet-18, showing that 4-bit quantization can achieve accuracy loss less than 1% on ImageNet.
- Learnable vs. Fixed λ: Experiments show that their learnable regularization coefficient consistently outperforms fixed coefficients across various low-precision settings for ResNet-18.
Limitations and Future Work
- Gradient Mismatch Still Exists: The authors acknowledge that their method still faces the gradient mismatch problem, albeit significantly reduced by their regularization approach.
- Optimization Complexity: While their method is simpler than some prior relaxation techniques, optimizing the learnable regularization coefficient adds a new parameter to tune.
- Future Work: The paper implicitly suggests exploring more aggressive quantization levels (e.g., 1-bit weights and activations) and applying their techniques to a wider range of network architectures and tasks. The effectiveness of their pipeline for both classification and regression tasks hints at broad applicability.
Summary updated