Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections

Xichuan Zhou, Shengli Li, A. K. Qin, Kunping Li, Fang Tang, Shengdong Hu, Shujun Liu, Zhi Lin

Abstract

Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data. However, implementing a deep neural network in embedded systems is a challenging task, because a typical deep neural network, such as a Deep Belief Network using 128x128 images as input, could exhaust Giga bytes of memory and result in bandwidth and computing bottleneck. To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections. The proposed method adaptively reduces the weights associated with negligible features to zero, leading to sparse feedforward network architecture. Furthermore, since the small proportion of important weights are significantly larger than zero, they can be robustly thresholded and represented using single-bit integers (-1 and +1), leading to implementations of deep neural networks with sparse and binary connections. Our experiments showed that, for the application of recognizing MNIST handwritten digits, the features extracted by a two-layer Deep Adaptive Network with about 25% reserved important connections achieved 97.2% classification accuracy, which was almost the same with the standard Deep Belief Network (97.3%). Furthermore, for efficient hardware implementations, the sparse-and-binary-weighted deep neural network could save about 99.3% memory and 99.9% computation units without significant loss of classification accuracy for pattern recognition applications.

Introduction: The Hardware Bottleneck

Deep Neural Networks (DNNs) have revolutionized pattern recognition, yet their deployment in embedded systems remains constrained by significant resource requirements. A standard Deep Belief Network (DBN) processing high-resolution images can demand gigabytes of memory and billions of floating-point operations, creating a bottleneck for power-sensitive devices like smart cameras or wearable technology. While hardware accelerators like FPGAs offer some relief, they often face challenges with high-dimensional data or accuracy degradation during low-precision quantization. This research proposes a "hardware-aware" learning algorithm, the Deep Adaptive Network (DAN), designed to optimize neural connections for efficiency from the ground up.

Methodology: Adaptive Restricted Boltzmann Machines

The DAN is constructed by stacking Adaptive Restricted Boltzmann Machines (AdaRBMs). The architecture is optimized through a two-step process:

  1. Mixed-Norm Regularization: During training, a mixed $L_{2,1}$ norm penalty is added to the objective function. Unlike standard $L_1$ or $L_2$ regularization, this mixed norm encourages group sparsity, effectively identifying and pruning entire rows or columns of the weight matrix that contribute minimally to the output.
  2. Robust Binarization: The training process causes weights to naturally cluster into three distinct groups: large negative, near-zero, and large positive. This allows for robust thresholding where near-zero weights are discarded, and the remaining significant weights are converted to single-bit integers (-1 and +1).
Comparison of DBN and DAN architectures showing the transition from fully connected to sparse layers.

Key Insights: Sparsity and Binarization

The DAN approach treats sparsity as a form of adaptive feature selection. By "muting" connections associated with negligible features, the network focuses its limited resources on the most discriminative information. A key finding is the "Three-Group" phenomenon, where weights naturally segregate, making binarization far more stable than in traditional networks where forced quantization often leads to catastrophic accuracy loss.

Weight parameters of DAN vs. DBN, highlighting the sparse and binary nature of the DAN connections.

Furthermore, the transition from floating-point multipliers to binary weights allows hardware implementations to replace complex arithmetic units with simple logic gates (such as XOR) and accumulators. This shift significantly reduces the silicon area and power consumption required for inference.

Experimental Results: MNIST Benchmark

The researchers evaluated the DAN using the MNIST handwritten digit dataset. The results demonstrate that extreme efficiency does not require sacrificing performance:

  • Accuracy: A two-layer DAN with only 25% of its connections reserved achieved 97.2% accuracy, nearly matching the 97.3% achieved by a full-precision DBN.
  • Memory Efficiency: The sparse-binary DAN ($DAN_b$) required only 30 KB of memory, a 99.3% reduction from the 4,950 KB needed for a standard DBN.
  • Computational Savings: The combination of sparsity and binary weights resulted in a 99.9% reduction in the complexity of computational units.
  • Robustness: Even when both weights and features were binarized ($DAN_B$), the network maintained a respectable 93.3% accuracy.

Conclusion and Future Directions

The Deep Adaptive Network provides a viable path for deploying sophisticated deep learning models on resource-constrained hardware. While the results on MNIST are promising, future work should explore the scalability of this mixed-norm approach on more complex datasets like ImageNet. Additionally, further research into fully quantizing the final classification layer and optimizing the training overhead of the mixed-norm regularization will be essential for creating end-to-end hardware-efficient pipelines.

Summary updated

Abstract

Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data. However, implementing a deep neural network in embedded systems is a challenging task, because a typical deep neural network, such as a Deep Belief Network using 128x128 images as input, could exhaust Giga bytes of memory and result in bandwidth and computing bottleneck. To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections. The proposed method adaptively reduces the weights associated with negligible features to zero, leading to sparse feedforward network architecture. Furthermore, since the small proportion of important weights are significantly larger than zero, they can be robustly thresholded and represented using single-bit integers (-1 and +1), leading to implementations of deep neural networks with sparse and binary connections. Our experiments showed that, for the application of recognizing MNIST handwritten digits, the features extracted by a two-layer Deep Adaptive Network with about 25% reserved important connections achieved 97.2% classification accuracy, which was almost the same with the standard Deep Belief Network (97.3%). Furthermore, for efficient hardware implementations, the sparse-and-binary-weighted deep neural network could save about 99.3% memory and 99.9% computation units without significant loss of classification accuracy for pattern recognition applications.
BibTeX
@misc{zhou2016deep,
  author = {Xichuan Zhou and Shengli Li and A. K. Qin and Kunping Li and Fang Tang and Shengdong Hu and Shujun Liu and Zhi Lin},
  title = {{Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections}},
  year = {2016},
  eprint = {1604.06154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.06154},
}
Figure