Abstract
In recent years, machine learning techniques based on neural networks for mobile computing have become increasingly popular. Classical multi-layer neural networks require matrix multiplications at each stage, an operation that is not energy efficient and drains mobile device batteries. We propose a new energy efficient neural network with the universal approximation property over the space of Lebesgue integrable functions. This "additive neural network" is based on a novel vector product definition, the ef-operator, which permits a multiplier-free implementation. In the ef-operation, the product of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product. Experiments on the MNIST dataset show that the classification performance of additive neural networks is comparable to traditional multi-layer perceptrons and convolutional neural networks (LeNet).
Introduction: The "Multiplication Tax"
Modern Deep Learning relies heavily on matrix multiplications. In hardware, multiplication operations consume significantly more power and silicon area than simple additions. This "multiplication tax" hinders the deployment of sophisticated AI models on battery-constrained edge devices like smartphones and wearables. While industry efforts often focus on specialized hardware or model pruning, this research addresses the problem at its mathematical foundation: redesigning the fundamental neuron to operate without multiplications.
Methodology: The ef-operator
The core innovation is the ef-operator ($\diamond$), which replaces the standard inner product $\sum (x_i \cdot w_i)$ with an addition-based alternative: $$x \diamond y := \sum_{i=1}^d \text{sign}(x_i \times y_i)(|x_i| + |y_i|)$$
This operator works by determining the sign through a trivial bitwise XOR and adding the absolute values of inputs and weights instead of multiplying them. Crucially, the authors prove that this structure maintains the Universal Approximation Property, ensuring that additive networks can theoretically represent any complex function just as well as traditional ones.
Optimization Landscapes
The transition from multiplication to addition changes the cost surface of the network. Because the ef-operator utilizes a sign function, the cost surface is divided into "hyperoctants." During training, the network can "jump" between these regions, presenting unique optimization challenges compared to the smooth surfaces of traditional networks.
Experimental Results
The researchers evaluated the Additive Neural Network against traditional "c-operator" models using the XOR problem and the MNIST dataset.
XOR and MNIST Performance
The additive network successfully solved the non-linear XOR problem with 100% accuracy. On the MNIST dataset, the performance gap was minimal:
- LeNet-5: Traditional (99.29%) vs. Additive (98.60%).
- MLP: The additive version remained within 0.034% to 0.7% of traditional models across various configurations.
Conclusion and Future Work
By replacing multiplications with additions, the proposed architecture significantly reduces the computational energy footprint for inference. While the "hyperoctant" nature of the cost function requires careful learning rate tuning, the results demonstrate that the $l_1$ norm (induced by the ef-operator) is a viable foundation for deep learning. Future work will focus on scaling these multiplier-free architectures to larger models like ResNets and Transformers.
Reference: Afrasiyabi, A., et al. "Energy Saving Additive Neural Network." arXiv:1702.02676v1 [cs.NE].
Summary updated