Redistribution of Weights and Activations for AdderNet Quantization

Ying Nie, Kai Han, Haikang Diao, Chuanjian Liu, Enhua Wu, Yunhe Wang

Abstract

Adder Neural Networks (AdderNets) significantly reduce energy consumption by replacing power-hungry multiplications with simple additions. However, they are notoriously difficult to quantize to low-bit integers because the mathematical laws that allow flexible scaling in standard CNNs do not apply. This paper introduces a novel "redistribution" strategy—using clustered group scales and a lossless weight-clamping trick—to achieve state-of-the-art 4-bit quantization accuracy, outperforming previous methods by over 8%.

Introduction: The Quantization Deadlock in AdderNets

Standard Convolutional Neural Networks (CNNs) rely on cross-correlation, which involves massive amounts of floating-point multiplications. AdderNets replace these with the $\ell_1$-norm (absolute difference), turning multiplications into additions. While this is great for hardware efficiency, deploying these models on edge devices requires quantization—converting weights and activations from 32-bit floats to low-bit integers like 4-bit or 8-bit.

In standard CNNs, the commutative law of multiplication allows weights ($W$) and activations ($X$) to be scaled independently ($s_w W \cdot s_x X = s_w s_x (W \cdot X)$). In AdderNets, this mathematical flexibility is lost: $|s_x X - s_w W| \neq s_x s_w |X - W|$. Consequently, previous researchers were forced to use a single "shared scale" for both weights and activations. This creates a "Goldilocks" problem: if the scale is tuned to the activations, large weights get "over-clamped" (truncated), destroying information. If the scale is tuned to the weights, small activations suffer from "bits waste," where the quantization is too coarse to represent subtle differences.

Comparison of quantization challenges in Vanilla vs. Adder Convolutions. AdderNets face a trade-off between over-clamping weights and wasting bits on activations.

Proposed Methodology: The Redistribution Strategy

The authors propose three main techniques to break this deadlock and redistribute the value ranges effectively:

1. Clustering-based Weights Grouping

Instead of one scale for the entire layer, the authors use $k$-means clustering to group weight kernels with similar ranges. Each group gets its own scale. This allows for finer granularity with negligible computational overhead.

2. Lossless Range Clamp for Weights

This is the paper's most clever mathematical contribution. When weight ranges exceed activation ranges, the authors clamp the weights to match the activations. Crucially, they prove that the mathematical difference caused by this clamping can be moved into the bias term of the layer. This makes the clamping "lossless"—the output remains identical to the original full-precision operation.

3. Outliers Clamp for Activations

To handle the "bits waste" issue, they use a simple hyperparameter-controlled strategy to prune extreme activation outliers, ensuring the available bit-range is used for the most frequent values.

The proposed redistribution workflow: (a) clustering weights, (b) clamping activation outliers, (c) lossless weight clamping with bias compensation, and (d) final quantization.

Key Results

The proposed method, called AdderQuant, shows significant improvements over the previous state-of-the-art (QSSF):

  • ImageNet Accuracy: On a 4-bit post-training quantized (PTQ) ResNet-18, AdderQuant achieved 66.5% top-1 accuracy. This is a massive 8.5% improvement over previous AdderNet quantization methods.
  • Near-Lossless 8-bit: At 8-bit quantization, the accuracy drop compared to full-precision models is almost zero (less than 0.2%).
  • Energy Efficiency: The authors demonstrate that their 4-bit AdderNet consumes significantly less energy than a standard 4-bit CNN while maintaining competitive accuracy, making it a viable candidate for ultra-low-power AI hardware.

Discussion and Limitations

While 4-bit results are impressive, the accuracy still drops significantly as you move toward 3-bit or 2-bit quantization. The authors note that Quantization-Aware Training (QAT) becomes necessary at these levels, which is more computationally expensive than Post-Training Quantization (PTQ). Furthermore, while the theory shows energy savings, the "group-shared scales" and "bias compensation" require specific hardware support to realize the theoretical efficiency gains in practice. Finally, the authors briefly mention that while low-power AI is good for the environment, it also makes surveillance devices easier to deploy, raising potential privacy concerns.

Summary updated

BibTeX
@inproceedings{nie2022redistribution,
  author = {Ying Nie and Kai Han and Haikang Diao and Chuanjian Liu and Enhua Wu and Yunhe Wang},
  title = {{Redistribution of Weights and Activations for AdderNet Quantization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=GMZOU32fp4},
}
Figure