Abstract
This paper proposes a new quantization framework on activation function of deep neural networks (DNN). We implement fixed-point DNN by quantizing the activations into powers-of-two integers. The costly multiplication operations in using DNN can be replaced with low-cost bit-shifts to massively save computations. Thus, applying DNN-based speech recognition on embedded systems becomes much easier. Experiments show that the proposed method leads to no performance degradation.
Introduction
Deep Neural Networks (DNNs) have revolutionized speech recognition, but their deployment on embedded systems and mobile devices is hindered by a heavy "computational tax." Standard DNNs perform millions of multiplications—specifically, weighted sums of inputs at every neuron—which are expensive in terms of energy and hardware area on FPGAs or specialized ICs. While existing techniques like pruning or low-rank factorization help reduce parameters, they often degrade accuracy. This research introduces a novel quantization framework that replaces power-hungry multiplications with low-cost bit-shifts without sacrificing performance.
Methods: Piecewise Quantized Sigmoid
The core innovation is a piecewise quantized sigmoidal activation function. In a standard DNN, the activation function (usually a sigmoid) outputs a continuous value between 0 and 1. The authors propose quantizing these outputs specifically into powers-of-two (e.g., $2^0, 2^{-1}, 2^{-2}, \dots, 2^{-(N-2)}$).
In digital logic, multiplying a number by a power-of-two is equivalent to a bit-shift operation, which is significantly faster and uses much less power than a full hardware multiplier. The implementation process involves:
- Linear Quantization of Weights: Weights are converted to 16-bit short integers. The authors found weights to be more sensitive to quantization than activations, so they maintained linear quantization for them.
- Piecewise Activation Quantization: Activations are mapped to the nearest power-of-two integer stage.
- Fine-Tuning: To adapt to the discretized activation landscape, the authors perform a single pass over the training data to fine-tune the weights.
Experimental Results
The researchers evaluated their method on a Chinese Large Vocabulary Continuous Speech Recognition (LVCSR) task using 3,000 hours of diverse speech data. The system was implemented on a Cyclone IV E FPGA.
Accuracy and Performance
Using $N=7$ quantization stages and fine-tuning, the model achieved a Word Error Rate (WER) of 18.3%. As shown in the table below, this matches the original high-precision floating-point model and outperforms standard 8-bit linear quantization (19.0%).
In terms of hardware efficiency, the piecewise quantized model reached 316 Frames Per Second (FPS), compared to only 109 FPS for the linearly quantized version—a 2.9x speedup. This improvement stems from eliminating the need for many dedicated hardware multipliers.
Key Insights and Conclusion
A critical finding of this work is the sensitivity asymmetry in DNNs: models are much more sensitive to the quantization of weights than activations. By focusing aggressive power-of-two quantization only on activations, the authors avoided the accuracy drops seen in previous attempts. The sigmoid function's bounded range makes it particularly well-suited for this mapping.
While the current work focuses on sigmoidal activations, the principles of replacing multiplications with bit-shifts offer a promising path for making deep learning more accessible on low-power hardware. Future research may explore applying similar techniques to unbounded functions like ReLU or utilizing quantization-aware training from scratch.
Summary updated