Energy Saving Additive Neural Network

Arman Afrasiyabi, Ozan Yildiz, Baris Nasir, Fatos T. Yarman-Vural, A. Enis ÇetinarXiv 2017 · arXiv · OpenReview

Abstract

In recent years, machine learning techniques based on neural networks for mobile computing have become increasingly popular. Classical multi-layer neural networks require matrix multiplications at each stage, an operation that is not energy efficient and drains mobile device batteries. We propose a new energy efficient neural network with the universal approximation property over the space of Lebesgue integrable functions. This "additive neural network" is based on a novel vector product definition, the ef-operator, which permits a multiplier-free implementation. In the ef-operation, the product of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product. Experiments on the MNIST dataset show that the classification performance of additive neural networks is comparable to traditional multi-layer perceptrons and convolutional neural networks (LeNet).

Introduction: The "Multiplication Tax"

Modern Deep Learning relies heavily on matrix multiplications. In hardware, multiplication operations consume significantly more power and silicon area than simple additions. This "multiplication tax" hinders the deployment of sophisticated AI models on battery-constrained edge devices like smartphones and wearables. While industry efforts often focus on specialized hardware or model pruning, this research addresses the problem at its mathematical foundation: redesigning the fundamental neuron to operate without multiplications.

Methodology: The ef-operator

The core innovation is the ef-operator ($\diamond$), which replaces the standard inner product $\sum (x_i \cdot w_i)$ with an addition-based alternative: $$x \diamond y := \sum_{i=1}^d \text{sign}(x_i \times y_i)(|x_i| + |y_i|)$$

This operator works by determining the sign through a trivial bitwise XOR and adding the absolute values of inputs and weights instead of multiplying them. Crucially, the authors prove that this structure maintains the Universal Approximation Property, ensuring that additive networks can theoretically represent any complex function just as well as traditional ones.

Optimization Landscapes

The transition from multiplication to addition changes the cost surface of the network. Because the ef-operator utilizes a sign function, the cost surface is divided into "hyperoctants." During training, the network can "jump" between these regions, presenting unique optimization challenges compared to the smooth surfaces of traditional networks.

Comparison of cost convergence between the traditional c-operator (a) and the proposed ef-operator (b).

Experimental Results

The researchers evaluated the Additive Neural Network against traditional "c-operator" models using the XOR problem and the MNIST dataset.

XOR and MNIST Performance

The additive network successfully solved the non-linear XOR problem with 100% accuracy. On the MNIST dataset, the performance gap was minimal:

  • LeNet-5: Traditional (99.29%) vs. Additive (98.60%).
  • MLP: The additive version remained within 0.034% to 0.7% of traditional models across various configurations.
Classification accuracy of an Additive Neural Network (ef-operator) with 3 hidden layers on MNIST across different learning rates.

Conclusion and Future Work

By replacing multiplications with additions, the proposed architecture significantly reduces the computational energy footprint for inference. While the "hyperoctant" nature of the cost function requires careful learning rate tuning, the results demonstrate that the $l_1$ norm (induced by the ef-operator) is a viable foundation for deep learning. Future work will focus on scaling these multiplier-free architectures to larger models like ResNets and Transformers.

Reference: Afrasiyabi, A., et al. "Energy Saving Additive Neural Network." arXiv:1702.02676v1 [cs.NE].

Summary updated

Abstract

In recent years, machine learning techniques based on neural networks for mobile computing become increasingly popular. Classical multi-layer neural networks require matrix multiplications at each stage. Multiplication operation is not an energy efficient operation and consequently it drains the battery of the mobile device. In this paper, we propose a new energy efficient neural network with the universal approximation property over space of Lebesgue integrable functions. This network, called, additive neural network, is very suitable for mobile computing. The neural structure is based on a novel vector product definition, called ef-operator, that permits a multiplier-free implementation. In ef-operation, the "product" of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product of the numbers. This "product" is used to construct a vector product in $R^N$. The vector product induces the $l_1$ norm. The proposed additive neural network successfully solves the XOR problem. The experiments on MNIST dataset show that the classification performances of the proposed additive neural networks are very similar to the corresponding multi-layer perceptron and convolutional neural networks (LeNet).
BibTeX
@misc{afrasiyabi2017energy,
  author = {Arman Afrasiyabi and Ozan Yildiz and Baris Nasir and Fatos T. Yarman-Vural and A. Enis Çetin},
  title = {{Energy Saving Additive Neural Network}},
  year = {2017},
  eprint = {1702.02676},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.02676},
}
Figure