Abstract
The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduce implementation complexity. We implement the entire training procedure in the log-domain, with fixed-point data representations. This training procedure is inspired by hardware-friendly approximations of log-domain addition which are based on look-up tables and bit-shifts. We show that our 16-bit log-based training can achieve classification accuracy within approximately 1% of the equivalent floating-point baselines for a number of commonly used datasets.
Introduction
Training modern deep neural networks (DNNs) is a computationally intensive process, often referred to as a "silicon tax." In standard hardware architectures, the most frequent and energy-consuming operation is the Multiply-Accumulate (MAC). While addition is relatively inexpensive in terms of circuit area and power, multiplication is significantly more costly. This burden restricts the feasibility of training models directly on edge devices, such as smartphones or IoT sensors, and contributes to the massive energy footprint of data centers.
While previous research into the Logarithmic Number System (LNS) has primarily focused on inference, this paper addresses the more significant challenge of end-to-end training. By maintaining data in the log-domain throughout the entire training process, the overhead of constant conversion between linear and logarithmic domains is eliminated.
Methodology
The proposed framework represents every numerical value $v$ by its sign and its logarithm: $V = \log_2(|v|)$. This transformation fundamentally changes the nature of neural network computations:
- Multiplication as Addition: In the log-domain, the product $x \times y$ is computed as the sum $X + Y$. This shift eliminates the need for complex and power-hungry multiplier circuits.
- Log-Domain Addition: The primary challenge in LNS is addition. Adding two numbers $x$ and $y$ in the log-domain requires calculating a correction term: $\max(X, Y) + \Delta(d)$, where $d = |X - Y|$.
- Hardware-Friendly Approximations: To maintain hardware simplicity, the complex $\Delta(d)$ function is approximated using two methods:
- Look-Up Tables (LUTs): Small tables storing pre-calculated values of the correction term.
- Bit-Shifts: An even simpler approximation that treats the correction as a power-of-two shift, enabling "multiplication-free" hardware.
The researchers also adapted standard DNN components for the log-domain, introducing the "Log-Leaky ReLU" (llReLU) activation function and a specialized log-domain Soft-max layer.
Experimental Results
The approach was evaluated using four benchmark image datasets: MNIST, Fashion-MNIST (FMNIST), EMNIST-Digits, and EMNIST-Letters. The experiments compared 12-bit and 16-bit log-domain training against standard floating-point and linear fixed-point baselines.
A key finding is that DNNs are remarkably robust to the noise introduced by these approximations. Coarse approximations, such as a 20-element LUT, were sufficient for most of the network. However, the Soft-max layer proved more sensitive, requiring a higher-resolution LUT (640 elements) to maintain accuracy.
On the MNIST dataset, the 16-bit log-domain model with a LUT achieved an accuracy of 97.2%, nearly matching the 97.4% accuracy of the floating-point baseline. Even the ultra-simple bit-shift approximation reached 96.5% accuracy. These results suggest that 16-bit log-domain training is highly effective in practice, despite theoretical concerns regarding precision loss.
Conclusion and Future Work
This research demonstrates that end-to-end DNN training in the log-domain is not only possible but also highly efficient. By replacing multipliers with adders and utilizing tiny LUTs, the authors estimate a potential 3x improvement in the area-delay product compared to traditional hardware designs.
Future work will involve testing this approach on more complex architectures like Convolutional Neural Networks (CNNs) and Transformers. Additionally, the researchers plan to move from software simulations to physical VLSI implementations to empirically measure power savings and hardware efficiency.
Summary updated