Stochastic Layer-Wise Precision in Deep Neural Networks

Griffin Lacey, Graham W. Taylor, Shawki Areibi

Introduction

The deployment of deep neural networks on resource-constrained edge devices, such as smartphones and embedded sensors, is often limited by high memory and computational requirements. Quantization—the process of reducing weight and activation precision from 32-bit floating point to low-bit integers—has emerged as a primary solution. However, traditional approaches typically employ uniform precision, assigning the same bit-width to every layer. This is inherently suboptimal, as different layers exhibit varying sensitivities to quantization noise.

Manually determining the optimal bit-width for each layer is a complex combinatorial problem. Furthermore, because bit-widths are discrete, they are traditionally incompatible with gradient-based optimization. This paper introduces a novel "precision allocation layer" that enables a network to automatically learn optimal, heterogeneous bit-widths under a fixed total bit budget.

Methodology

The proposed framework allows a network to stochastically explore different precision configurations during the training phase.

Precision Allocation Layer

A dedicated layer manages a global "bit budget." For instance, in a 5-layer network with a 10-bit budget, this layer determines how to distribute those bits across the architecture.

Differentiable Sampling with Gumbel-Softmax

To overcome the non-differentiability of discrete bit-width choices, the authors utilize the Gumbel-Softmax (or Concrete) distribution. This "trick" allows the model to sample precision configurations while maintaining a differentiable path for backpropagation.

Temperature Annealing

The training process relies on an annealing schedule. Early in training, a high "temperature" ($\tau$) encourages broad exploration of the precision space, functioning similarly to Dropout. As training progresses, the temperature is lowered, causing the distribution to "harden" into a fixed, deterministic bit-width assignment for each layer.

Evolution of precision allocation from early to late training via temperature annealing.

Experimental Setup

The methodology was evaluated on two benchmark computer vision tasks:

  • MNIST: Utilizing a 5-layer LeNet-style architecture to classify handwritten digits.
  • ImageNet (ILSVRC12): Utilizing the AlexNet architecture on a large-scale dataset of 1.2 million images.

The framework was implemented on top of DoReFa-Net, a standard method for training low-bit networks.

Key Results and Insights

Accuracy Improvements

The learned precision approach consistently outperformed uniform baselines. On MNIST, a model with a 10-bit budget (averaging 2 bits per layer) achieved a 2.32% error rate, a significant improvement over the 3.73% error of a standard uniform 2-bit model. Similar gains were observed on ImageNet; with a 28-bit budget, the learned model reached a 47.69% top-1 error, surpassing the uniform 4-bit baseline of 47.99%.

Validation error on MNIST comparing learned precision against uniform bit-width baselines.

Precision as a Regularizer

A key discovery was the regularization effect of stochastic precision. Even in cases where the final learned allocation was nearly uniform, the act of varying precisions during training improved generalization. This "precision dropout" helps the model become more robust to noise.

Heterogeneity and Hardware Efficiency

The model naturally learns to prioritize certain layers. For example, on MNIST, the network favored higher precision for later layers. By maintaining a fixed total bit budget, the authors ensure predictable memory and speed characteristics. Using custom CUDA kernels (bit-GEMM), they verified that configurations with 16 total bits or fewer provide substantial GPU speedups over standard 32-bit implementations on modern hardware like the NVIDIA Tesla V100.

Discussion and Future Work

While highly effective for weights and activations, the authors found that learning precision for gradients was less beneficial. Future research may explore "dynamic" or "per-example" precision, where the network adjusts its bit-depth based on the complexity of the input image. Although the training process requires careful tuning of the annealing schedule, this work successfully moves quantization from a manual hyperparameter tuning task into an automated, differentiable part of the training loop.

Summary updated

Abstract

Low precision weights, activations, and gradients have been proposed as a way to improve the computational efficiency and memory footprint of deep neural networks. Recently, low precision networks have even shown to be more robust to adversarial attacks. However, typical implementations of low precision DNNs use uniform precision across all layers of the network. In this work, we explore whether a heterogeneous allocation of precision across a network leads to improved performance, and introduce a learning scheme where a DNN stochastically explores multiple precision configurations through learning. This permits a network to learn an optimal precision configuration. We show on convolutional neural networks trained on MNIST and ILSVRC12 that even though these nets learn a uniform or near-uniform allocation strategy respectively, stochastic precision leads to a favourable regularization effect improving generalization.
BibTeX
@inproceedings{lacey2018stochastic,
  author = {Griffin Lacey and Graham W. Taylor and Shawki Areibi},
  title = {{Stochastic Layer-Wise Precision in Deep Neural Networks}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=ELRA_Gwy_hn},
}
Figure