CoNLoCNN: Exploiting Correlation and Non-Uniform Quantization for Energy-Efficient Low-precision Deep Convolutional Neural Networks

Muhammad Abdullah Hanif, Giuseppe Maria Sarda, Alberto Marchisio, Guido Masera, Maurizio Martina, Muhammad Shafique

Abstract

In today's era of smart cyber-physical systems, Deep Neural Networks (DNNs) have become ubiquitous due to their state-of-the-art performance in complex real-world applications. The high computational complexity of these networks, which translates to increased energy consumption, is the foremost obstacle towards deploying large DNNs in resource-constrained systems. Fixed-Point (FP) implementations achieved through post-training quantization are commonly used to curtail the energy consumption of these networks. However, the uniform quantization intervals in FP restrict the bit-width of data structures to large values due to the need to represent most of the numbers with sufficient resolution and avoid high quantization errors. In this paper, we leverage the key insight that (in most of the scenarios) DNN weights and activations are mostly concentrated near zero and only a few of them have large magnitudes. We propose CoNLoCNN, a framework to enable energy-efficient low-precision deep convolutional neural network inference by exploiting: (1) non-uniform quantization of weights enabling simplification of complex multiplication operations; and (2) correlation between activation values enabling partial compensation of quantization errors at low cost without any run-time overheads. To significantly benefit from non-uniform quantization, we also propose a novel data representation format, Encoded Low-Precision Binary Signed Digit, to compress the bit-width of weights while ensuring direct use of the encoded weight for processing using a novel multiply-and-accumulate (MAC) unit design.

Introduction

Deploying state-of-the-art Deep Neural Networks (DNNs) on edge devices—such as smartphones or IoT sensors—presents a significant challenge in balancing accuracy and energy efficiency. While quantization is a standard approach to reduce computational overhead, it often necessitates expensive retraining or results in substantial accuracy degradation when precision drops below 8 bits. Standard Fixed-Point (FP) arithmetic utilizes uniform quantization intervals, which are inefficient for DNN weights that typically exhibit a "long-tailed" distribution clustered around zero.

CoNLoCNN (Correlation and Non-Linear Quantization for CNNs) addresses these limitations through a post-training quantization (PTQ) framework. By rethinking weight representation and error handling, CoNLoCNN achieves significant energy savings without requiring access to the original training pipeline or datasets.

Methodology

The CoNLoCNN framework integrates four primary components to optimize DNN inference, as illustrated in the system overview.

Overview of the CoNLoCNN framework components.

Non-Uniform Quantization and ELP_BSD

The core of the framework is the Encoded Low-Precision Binary Signed Digit (ELP_BSD) format. Unlike standard binary, weights are represented as a sum of power-of-two digits (e.g., $2^{-2} + 2^{-5}$). This non-uniform representation aligns with the natural distribution of DNN weights, providing higher resolution where values are most frequent (near zero) and lower resolution for rare, large-magnitude outliers.

Comparison between uniform and non-uniform quantization levels relative to the weight distribution.

Multiplier-Free Hardware Design

By representing weights as sums of powers-of-two, the hardware can replace traditional, energy-intensive multipliers with barrel shifters and adders. This specialized computational array significantly reduces silicon area and power consumption during inference.

Spatial Error Compensation

CoNLoCNN leverages the spatial correlation inherent in image data. Neighboring pixels in a feature map often share similar values. The framework employs a compile-time algorithm to adjust quantized weights within a filter such that their individual quantization errors partially cancel each other out during convolution. This compensation occurs entirely offline, introducing zero runtime overhead.

Key Results

The framework was evaluated using ImageNet benchmarks on AlexNet and VGG-16 architectures. Key findings include:

  • Energy Efficiency: For AlexNet, CoNLoCNN achieved a 76% reduction in Power-Delay Product (PDP) compared to conventional 8-bit designs.
  • Accuracy Preservation: This massive energy saving came at a minimal cost, with only a 1.44% drop in Top-1 accuracy for AlexNet.
  • Competitive Performance: At ultra-low bit-widths, CoNLoCNN outperformed other state-of-the-art approximate computing methods, such as CAxCNN, by approximately 4.5% on ImageNet.

Conclusion

CoNLoCNN demonstrates that by co-designing hardware and quantization algorithms to exploit the statistical properties of DNNs, it is possible to achieve dramatic energy savings in post-training scenarios. While the current error compensation strategy relies on spatial correlation—making it ideal for convolutional layers—future work may extend these principles to fully connected layers or non-spatial data. The integration of this framework with model pruning techniques offers a promising path toward ultra-efficient edge AI deployment.

Summary updated

Abstract

In today's era of smart cyber-physical systems, Deep Neural Networks (DNNs) have become ubiquitous due to their state-of-the-art performance in complex real-world applications. The high computational complexity of these networks, which translates to increased energy consumption, is the foremost obstacle towards deploying large DNNs in resource-constrained systems. Fixed-Point (FP) implementations achieved through post-training quantization are commonly used to curtail the energy consumption of these networks. However, the uniform quantization intervals in FP restrict the bit-width of data structures to large values due to the need to represent most of the numbers with sufficient resolution and avoid high quantization errors. In this paper, we leverage the key insight that (in most of the scenarios) DNN weights and activations are mostly concentrated near zero and only a few of them have large magnitudes. We propose CoNLoCNN, a framework to enable energy-efficient low-precision deep convolutional neural network inference by exploiting: (1) non-uniform quantization of weights enabling simplification of complex multiplication operations; and (2) correlation between activation values enabling partial compensation of quantization errors at low cost without any run-time overheads. To significantly benefit from non-uniform quantization, we also propose a novel data representation format, Encoded Low-Precision Binary Signed Digit, to compress the bit-width of weights while ensuring direct use of the encoded weight for processing using a novel multiply-and-accumulate (MAC) unit design.
BibTeX
@inproceedings{hanif2022conlocnn,
  author = {Muhammad Abdullah Hanif and Giuseppe Maria Sarda and Alberto Marchisio and Guido Masera and Maurizio Martina and Muhammad Shafique},
  title = {{CoNLoCNN: Exploiting Correlation and Non-Uniform Quantization for Energy-Efficient Low-precision Deep Convolutional Neural Networks}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9892902},
}
Figure