Optimizing Information Theory Based Bitwise Bottlenecks for Efficient Mixed-Precision Activation Quantization

Xichuan Zhou, Kui Liu, Cong Shi, Haijun Liu, Ji LiuAAAI 2021 · DOI

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success across various domains, but their deployment on resource-constrained "edge" devices—such as smartphones and IoT sensors—remains a significant challenge due to high memory and computational demands. Quantization, the process of reducing the precision of network weights and activations (e.g., from 32-bit floating-point to low-bit integers), is a primary strategy for model compression.

However, many existing quantization methods rely on heuristics and often apply uniform bit-widths across all layers. This approach fails to account for the varying information density within different parts of a network. This paper, presented at AAAI-21, introduces a theoretically grounded "Bitwise Bottleneck" framework. By leveraging rate-distortion theory from information science, the authors propose a method to adaptively quantize activations bit-by-bit, maximizing efficiency while preserving model "intelligence."

The Bitwise Bottleneck Framework

The core innovation of this work is framing activation quantization as a rate-distortion problem. In communication theory, this involves finding the optimal balance between signal compression (rate) and signal quality (distortion).

Adaptive Quantization via Variable Coefficients

Traditional quantization represents a numerical value as a sum of bits multiplied by fixed powers of two ($2^0, 2^1, 2^2, \dots$). The Bitwise Bottleneck replaces these fixed constants with learnable, variable coefficients ($\alpha$).

The Bitwise Bottleneck quantization process, transforming floating-point data into optimized quantized data using variable coefficients.

As illustrated in Figure 2, the process involves three main steps:

  1. Initial Quantization: Floating-point activations are first mapped to an initial fixed-point representation.
  2. Sparse Optimization: The problem is formulated using $L_1$-norm-based sparse optimization. If a specific bit contributes little to the overall representation of the signal, its corresponding coefficient $\alpha$ is driven to zero.
  3. Bit Pruning: Bits with zero coefficients are effectively pruned, leading to a reduced bit-width for that specific layer.

This adaptive approach allows each layer to determine its own optimal precision based on a user-defined Peak Signal-to-Noise Ratio (PSNR) threshold. By allowing coefficients to vary, the model can better fit the actual distribution of activations, providing greater representational flexibility than standard fixed-point methods.

Experimental Evaluation

The researchers evaluated the Bitwise Bottleneck across three standard computer vision benchmarks:

  • MNIST: Handwritten digit recognition.
  • CIFAR-10: Small-scale image classification.
  • ImageNet (ILSVRC2012): Large-scale image classification using the ResNet50 architecture.

These datasets provide a comprehensive testing ground for the method's scalability from simple tasks to complex, real-world visual recognition.

Analysis of Bitwise Sparsity

A key insight from the study is the empirical validation of bitwise sparsity within neural networks. The authors observed that activations are not uniformly informative across their bit-representation.

Activation distributions and corresponding bitwise rates/coefficients across different network layers.

As shown in Figure 5, the "high-end" (most significant) bits are often statistically near-zero in certain layers, while "low-end" bits frequently contain mostly noise. The Bitwise Bottleneck automatically identifies these redundancies. By pruning uninformative bits, the method achieves mixed-precision quantization that is tailored to the specific informational needs of each layer. This confirms that classical signal processing tools can effectively manage the complex signals within deep learning models.

Key Results and Performance

The Bitwise Bottleneck demonstrates superior performance compared to existing state-of-the-art quantization techniques:

  • Accuracy: On ImageNet with ResNet50, the method achieved 75.7% Top-1 accuracy using 5-bit activations, outperforming popular methods like DSQ (75.2%) and DoReFa-Net (73.8%).
  • Efficiency: The approach yielded over 6x improvements in both memory footprint and computational efficiency compared to standard 32-bit floating-point models.
  • Latency: By utilizing bitwise layers before convolutions, the authors report that inference latency can be reduced by over 90%.
  • Robustness: Even when compressing activations to as low as 4 bits, the accuracy loss remained marginal (only 0.8% lower than the initial quantization), highlighting the robustness of the bottleneck approach.

Conclusion and Future Work

The Bitwise Bottleneck provides a rigorous, information-theoretic path toward highly efficient neural networks. By treating activations as signals to be compressed and utilizing sparse optimization to prune uninformative bits, the method achieves significant gains in efficiency without compromising accuracy.

While highly effective at exploiting existing sparsity, the authors note that the current method does not actively create sparsity during training. Future research will focus on incorporating sparse regularization directly into the training phase to induce even greater compressibility. Additionally, while the theoretical benefits are clear, the practical implementation of mixed-precision bitwise operations on standard hardware remains an ongoing engineering challenge for the community.

Summary updated

BibTeX
@inproceedings{zhou2021optimizing,
  author = {Xichuan Zhou and Kui Liu and Cong Shi and Haijun Liu and Ji Liu},
  title = {{Optimizing Information Theory Based Bitwise Bottlenecks for Efficient Mixed-Precision Activation Quantization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16474},
}
Figure