HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface

Sungho Shin, Jinhwan Park, Yoonho Boo, Wonyong SungAAAI 2020 · DOI

Introduction

Deep Neural Networks (DNNs) are increasingly deployed on edge devices, necessitating efficient compression techniques like quantization. Quantization reduces weight and activation precision (e.g., to 2-bit), significantly lowering memory and energy consumption. However, training ultra-low-bit Quantized Neural Networks (QDNNs) often results in significant accuracy degradation. This paper introduces HLHLp (High-Low-High-Low precision), a novel training scheme that leverages quantization noise to guide models toward "flat minima" in the loss surface, ensuring better generalization and performance.

The Challenge: Sharp vs. Flat Minima

Standard training for QDNNs frequently converges to "sharp minima"—regions where the loss function is highly sensitive to small weight perturbations. Because quantization inherently shifts weight values, a model in a sharp minimum will experience a dramatic increase in error once discretized.

To achieve robustness, a model must find a flat minimum, where the loss remains low even if weights are slightly nudged. The HLHLp method treats quantization error not as a hurdle, but as a form of stochastic noise that can be used to "shake" the model out of narrow, sharp valleys and into wider, flatter basins.

The HLHLp Training Scheme

The HLHLp method moves away from traditional single-pass fine-tuning, employing a four-stage cyclic approach:

  1. H-step (High-precision): Establishes a strong floating-point baseline.
  2. L-step (Low-precision Coarse-tuning): Retrains at 2-bit precision with a high learning rate. This stage uses aggressive quantization noise to escape local optima.
  3. H-step (High-precision Fine-tuning): Switches back to higher precision (e.g., 8-bit) with a reduced learning rate to center the model within a wide basin.
  4. L-step (Low-precision Fine-tuning): Final 2-bit quantization at a very low learning rate to settle into the optimal flat minimum.
3D visualization of the loss surface showing how HLHLp navigates toward a flatter minimum compared to standard floating-point and high-low precision baselines.

The innovation lies in the mathematical link between the learning rate ($\eta$) and quantization precision ($b$). By controlling the ratio $\eta/2^b$, researchers can tune the intensity of the search for flatter regions.

Experimental Validation

The authors evaluated HLHLp across image classification, language modeling, and speech recognition tasks.

Image Classification

On benchmarks like CIFAR-10 and CIFAR-100, HLHLp consistently outperformed standard fine-tuning methods across various architectures, including ResNet and MobileNetV2.

Performance comparison on CIFAR-10 and CIFAR-100 datasets. HLHLp consistently achieves higher accuracy and smaller gaps from the floating-point baseline compared to standard fine-tuning.

Recurrent Neural Networks (RNNs)

The method achieved state-of-the-art results for 2-bit RNNs. Notably, on the WikiText-2 dataset, an HLHLp-trained GRU outperformed its full-precision counterpart, demonstrating the powerful regularizing effect of the training scheme.

Robustness in Speech Recognition

In noisy environments (Aurora-4), HLHLp-trained models showed superior robustness compared to standard methods, confirming that flat minima lead to better real-world generalization.

Conclusion

HLHLp demonstrates that quantization noise, when strategically managed through alternating precision and learning rates, is a powerful tool for optimizing DNNs. While the multi-stage training increases computational cost, the gains in accuracy and robustness—especially for ultra-low-bit models—make it a compelling strategy for deploying high-performance AI on resource-constrained hardware.

Summary updated

BibTeX
@inproceedings{shin2020hlhlp,
  author = {Sungho Shin and Jinhwan Park and Yoonho Boo and Wonyong Sung},
  title = {{HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.6035},
}
Figure