Neural Network Training with Approximate Logarithmic Computations

Arnab Sanyal, Peter A. Beerel, Keith M. Chugg

Abstract

The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduce implementation complexity. We implement the entire training procedure in the log-domain, with fixed-point data representations. This training procedure is inspired by hardware-friendly approximations of log-domain addition which are based on look-up tables and bit-shifts. We show that our 16-bit log-based training can achieve classification accuracy within approximately 1% of the equivalent floating-point baselines for a number of commonly used datasets.

Introduction

Training modern deep neural networks (DNNs) is a computationally intensive process, often referred to as a "silicon tax." In standard hardware architectures, the most frequent and energy-consuming operation is the Multiply-Accumulate (MAC). While addition is relatively inexpensive in terms of circuit area and power, multiplication is significantly more costly. This burden restricts the feasibility of training models directly on edge devices, such as smartphones or IoT sensors, and contributes to the massive energy footprint of data centers.

While previous research into the Logarithmic Number System (LNS) has primarily focused on inference, this paper addresses the more significant challenge of end-to-end training. By maintaining data in the log-domain throughout the entire training process, the overhead of constant conversion between linear and logarithmic domains is eliminated.

Methodology

The proposed framework represents every numerical value $v$ by its sign and its logarithm: $V = \log_2(|v|)$. This transformation fundamentally changes the nature of neural network computations:

  1. Multiplication as Addition: In the log-domain, the product $x \times y$ is computed as the sum $X + Y$. This shift eliminates the need for complex and power-hungry multiplier circuits.
  2. Log-Domain Addition: The primary challenge in LNS is addition. Adding two numbers $x$ and $y$ in the log-domain requires calculating a correction term: $\max(X, Y) + \Delta(d)$, where $d = |X - Y|$.
  3. Hardware-Friendly Approximations: To maintain hardware simplicity, the complex $\Delta(d)$ function is approximated using two methods:
  4. Look-Up Tables (LUTs): Small tables storing pre-calculated values of the correction term.
  5. Bit-Shifts: An even simpler approximation that treats the correction as a power-of-two shift, enabling "multiplication-free" hardware.
Approximation of the log-addition correction function $\Delta(d)$ using bit-shifts and LUTs.

The researchers also adapted standard DNN components for the log-domain, introducing the "Log-Leaky ReLU" (llReLU) activation function and a specialized log-domain Soft-max layer.

Experimental Results

The approach was evaluated using four benchmark image datasets: MNIST, Fashion-MNIST (FMNIST), EMNIST-Digits, and EMNIST-Letters. The experiments compared 12-bit and 16-bit log-domain training against standard floating-point and linear fixed-point baselines.

A key finding is that DNNs are remarkably robust to the noise introduced by these approximations. Coarse approximations, such as a 20-element LUT, were sufficient for most of the network. However, the Soft-max layer proved more sensitive, requiring a higher-resolution LUT (640 elements) to maintain accuracy.

Validation accuracy comparison on MNIST and Fashion-MNIST datasets across different bit-widths and domains.

On the MNIST dataset, the 16-bit log-domain model with a LUT achieved an accuracy of 97.2%, nearly matching the 97.4% accuracy of the floating-point baseline. Even the ultra-simple bit-shift approximation reached 96.5% accuracy. These results suggest that 16-bit log-domain training is highly effective in practice, despite theoretical concerns regarding precision loss.

Conclusion and Future Work

This research demonstrates that end-to-end DNN training in the log-domain is not only possible but also highly efficient. By replacing multipliers with adders and utilizing tiny LUTs, the authors estimate a potential 3x improvement in the area-delay product compared to traditional hardware designs.

Future work will involve testing this approach on more complex architectures like Convolutional Neural Networks (CNNs) and Transformers. Additionally, the researchers plan to move from software simulations to physical VLSI implementations to empirically measure power savings and hardware efficiency.

Summary updated

Abstract

The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduce implementation complexity. We implement the entire training procedure in the log-domain, with fixed-point data representations. This training procedure is inspired by hardware-friendly approximations of log-domain addition which are based on look-up tables and bit-shifts. We show that our 16-bit log-based training can achieve classification accuracy within approximately 1% of the equivalent floating-point baselines for a number of commonly used datasets.
BibTeX
@inproceedings{sanyal2020neural,
  author = {Arnab Sanyal and Peter A. Beerel and Keith M. Chugg},
  title = {{Neural Network Training with Approximate Logarithmic Computations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=kvGAQbJBYczs},
}
Figure