Energy Efficient Hadamard Neural Networks

T. Ceren Deveci, Serdar Çakir, A. Enis ÇetinarXiv 2018 · arXiv · OpenReview

Abstract

Deep learning has made significant improvements at many image processing tasks in recent years, such as image classification, object recognition and object detection. Convolutional neural networks (CNN), which is a popular deep learning architecture designed to process data in multiple array form, show great success to almost all detection & recognition problems and computer vision tasks. However, the number of parameters in a CNN is too high such that the computers require more energy and larger memory size. In order to solve this problem, we propose a novel energy efficient model Binary Weight and Hadamard-transformed Image Network (BWHIN), which is a combination of Binary Weight Network (BWN) and Hadamard-transformed Image Network (HIN). It is observed that energy efficiency is achieved with a slight sacrifice at classification accuracy. Among all energy efficient networks, our novel ensemble model outperforms other energy efficient models.

Introduction

While Convolutional Neural Networks (CNNs) have revolutionized computer vision, their deployment on "edge" devices—such as smartphones, wearables, and IoT sensors—is hindered by high computational and energy requirements. The primary bottleneck is the billions of floating-point matrix multiplications required for inference. To address this, researchers are exploring "multiplier-less" architectures that replace expensive multiplications with simpler, energy-efficient operations like addition and subtraction.

Methodology: The BWHIN Architecture

The proposed Binary Weight and Hadamard-transformed Image Network (BWHIN) is an ensemble model that integrates two energy-saving strategies:

  1. Binary Weight Networks (BWN): Weights are approximated as either +1 or -1, converting convolutions into addition/subtraction operations.
  2. Hadamard-transformed Image Networks (HIN): Input images undergo a Fast Walsh-Hadamard Transform (FWHT). This transform compresses data using only +1 and -1 coefficients and requires zero multiplications.

The BWHIN model processes a "real" image through a BWN stream and a "Hadamard-transformed" version through an HIN stream. The outputs of these streams are averaged before the final classification layer.

The BWHIN ensemble architecture, featuring dual CNN streams for real and Hadamard-transformed images.

Experimental Setup

The model was evaluated using two architectural variants: ConvPool-CNN, which utilizes traditional max-pooling layers, and All-CNN, which replaces pooling with strided convolutions for improved efficiency. These were tested on the MNIST (grayscale digits) and CIFAR-10 (color images) datasets.

Detailed layer configurations for the ConvPool-CNN and All-CNN architectures used in the experiments.

Key Results and Insights

The BWHIN model demonstrates that significant energy savings can be achieved with manageable accuracy trade-offs:

  • Performance: BWHIN achieved 98.96% accuracy on MNIST (vs. 99.48% for standard CNN) and 72.65% on CIFAR-10 (vs. 82.64%).
  • Efficiency of Strided Convolutions: The "All-CNN" approach proved more efficient than pooling, as it avoids redundant calculations.
  • Ensemble Robustness: Combining the BWN and HIN streams allowed the network to recover accuracy lost during the binarization process, outperforming either sub-network used in isolation.

Conclusion and Future Work

BWHIN offers a promising path for deploying deep learning on low-power hardware. While an accuracy gap remains on complex datasets like CIFAR-10, future research will focus on scaling the architecture to larger datasets like ImageNet and exploring hardware-level integration of the Hadamard transform.

Summary updated

Abstract

Deep learning has made significant improvements at many image processing tasks in recent years, such as image classification, object recognition and object detection. Convolutional neural networks (CNN), which is a popular deep learning architecture designed to process data in multiple array form, show great success to almost all detection \& recognition problems and computer vision tasks. However, the number of parameters in a CNN is too high such that the computers require more energy and larger memory size. In order to solve this problem, we propose a novel energy efficient model Binary Weight and Hadamard-transformed Image Network (BWHIN), which is a combination of Binary Weight Network (BWN) and Hadamard-transformed Image Network (HIN). It is observed that energy efficiency is achieved with a slight sacrifice at classification accuracy. Among all energy efficient networks, our novel ensemble model outperforms other energy efficient models.
BibTeX
@misc{deveci2018energy,
  author = {T. Ceren Deveci and Serdar Çakir and A. Enis Çetin},
  title = {{Energy Efficient Hadamard Neural Networks}},
  year = {2018},
  eprint = {1805.05421},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.05421},
}
Figure