Crossbar-aware neural network pruning

Ling Liang, Lei Deng, Yueling Zeng, Xing Hu, Yu Ji, Xin Ma, Guoqi Li, Yuan Xie

Abstract: Crossbar architecture based devices have been widely adopted in neural network accelerators by taking advantage of the high efficiency on vector-matrix multiplication (VMM) operations. However, in the case of convolutional neural networks (CNNs), the efficiency is compromised dramatically due to the large amounts of data reuse. Although some mapping methods have been designed to achieve a balance between the execution throughput and resource overhead, the resource consumption cost is still huge while maintaining the throughput. Network pruning is a promising and widely studied leverage to shrink the model size. Whereas, previous work didn't consider the crossbar architecture and the corresponding mapping method, which cannot be directly utilized by crossbar-based neural network accelerators. Tightly combining the crossbar structure and its mapping, this paper proposes a crossbar-aware pruning framework based on a formulated L0-norm constrained optimization problem. Specifically, we design an L0-norm constrained gradient descent (LGD) with relaxant probabilistic projection (RPP) to solve this problem. Two grains of sparsity are successfully achieved: i) intuitive crossbar-grain sparsity and ii) column-grain sparsity with output recombination, based on which we further propose an input feature maps (FMs) reorder method to improve the model accuracy. We evaluate our crossbar-aware pruning framework on median-scale CIFAR10 dataset and large-scale ImageNet dataset with VGG and ResNet models. Our method is able to reduce the crossbar overhead by 44%-72% with little accuracy degradation. This work greatly saves the resource and the related energy cost, which provides a new co-design solution for mapping CNNs onto various crossbar devices with significantly higher efficiency.

The Challenge of Crossbar-based CNN Acceleration

Crossbar architectures, utilizing emerging technologies like RRAM, PCM, or specialized SRAM, are exceptionally efficient at performing Vector-Matrix Multiplication (VMM) in the analog domain. However, mapping Convolutional Neural Networks (CNNs) to this hardware is challenging due to massive data reuse. To maintain high throughput, compilers often "unfold" the network, duplicating weights across many physical crossbars. This leads to a massive "resource explosion."

Architecture of a crossbar-based neural network accelerator showing peripheral circuits like ADCs and DACs.

Traditional pruning methods (like element-wise or channel-wise pruning) do not solve this because they do not respect the physical boundaries of the hardware. If a crossbar contains even a few residual non-zero weights, the entire unit—including its power-hungry ADCs, DACs, and peripheral circuits—must remain active. To truly save energy and area, the hardware must be able to remove the entire crossbar unit.

Crossbar-Aware Pruning Framework

The proposed framework treats hardware constraints as first-class citizens during the optimization process by formulating pruning as an $L_0$-norm constrained optimization problem. Instead of using $L_1$ or $L_2$ regularization, which merely encourages small weights, the $L_0$ constraint strictly limits the number of active hardware units.

To solve this NP-hard problem, the authors developed $L_0$-norm constrained Gradient Descent (LGD) coupled with Relaxant Probabilistic Projection (RPP). RPP provides a probabilistic approach to selecting pruning candidates, helping the model converge to a better sparse solution than deterministic hard-thresholding.

Sparsity Grains and Recombination

The framework achieves two distinct grains of hardware sparsity:

  1. Crossbar-grain: Pruning entire crossbar units when all weights within them are deemed redundant.
  2. Column-grain with Recombination: Pruning specific columns within crossbars and then "recombining" the remaining non-zero columns from different units into a single, densely packed crossbar.
Mapping CNN operations to crossbars: (a) logical convolution, (c) mathematical decomposition, and (b) physical mapping with column recombination.

Additionally, an Input Feature Map (FM) Reordering strategy is used to group important channels together. This makes it easier for the LGD solver to prune away entire groups of less-important channels, further improving hardware efficiency without sacrificing accuracy.

Experimental Evaluation

The framework was evaluated on standard computer vision benchmarks using VGG and ResNet architectures. The results demonstrate significant hardware savings:

  • VGG8 (CIFAR-10): Achieved a massive 72.1% reduction in compute crossbars.
  • VGG16 (ImageNet): Reduced compute crossbars by 59.8% with minimal accuracy loss after fine-tuning.
  • ResNet18 (ImageNet): Reduced compute crossbars by 44.2%. ResNet proved more sensitive to pruning due to its already compact residual structure.

Conclusion

This hardware-software co-design approach demonstrates that pruning for hardware is not just about weight matrix sparsity, but about maximizing hardware utilization. By removing the peripheral circuits associated with pruned crossbars, the framework provides a direct path to reducing the physical footprint and power leakage of next-generation AI accelerators. Future work will look to address non-compute overheads, such as data buffering and activation reduction units, to further optimize the entire system.

Summary updated

Abstract

Crossbar architecture based devices have been widely adopted in neural network accelerators by taking advantage of the high efficiency on vector-matrix multiplication (VMM) operations. However, in the case of convolutional neural networks (CNNs), the efficiency is compromised dramatically due to the large amounts of data reuse. Although some mapping methods have been designed to achieve a balance between the execution throughput and resource overhead, the resource consumption cost is still huge while maintaining the throughput. Network pruning is a promising and widely studied leverage to shrink the model size. Whereas, previous work didn`t consider the crossbar architecture and the corresponding mapping method, which cannot be directly utilized by crossbar-based neural network accelerators. Tightly combining the crossbar structure and its mapping, this paper proposes a crossbar-aware pruning framework based on a formulated L0-norm constrained optimization problem. Specifically, we design an L0-norm constrained gradient descent (LGD) with relaxant probabilistic projection (RPP) to solve this problem. Two grains of sparsity are successfully achieved: i) intuitive crossbar-grain sparsity and ii) column-grain sparsity with output recombination, based on which we further propose an input feature maps (FMs) reorder method to improve the model accuracy. We evaluate our crossbar-aware pruning framework on median-scale CIFAR10 dataset and large-scale ImageNet dataset with VGG and ResNet models. Our method is able to reduce the crossbar overhead by 44%-72% with little accuracy degradation. This work greatly saves the resource and the related energy cost, which provides a new co-design solution for mapping CNNs onto various crossbar devices with significantly higher efficiency.
BibTeX
@inproceedings{liang2018crossbar,
  author = {Ling Liang and Lei Deng and Yueling Zeng and Xing Hu and Yu Ji and Xin Ma and Guoqi Li and Yuan Xie},
  title = {{Crossbar-aware neural network pruning}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=TiGfE7zjSdg},
}
Figure