Crossbar-Aware Neural Network Pruning

Ling Liang, Lei Deng, Yueling Zeng, Xing Hu, Yu Ji, Xin Ma, Guoqi Li, Yuan Xie

TL;DR

This paper introduces a hardware-aware pruning framework specifically designed for crossbar-based neural network accelerators. By formulating pruning as an $L_0$-norm constrained optimization problem and introducing "column-grain" sparsity with hardware recombination, the authors reduce crossbar overhead by 44%–72% with negligible loss in accuracy.

What Problem Does This Paper Address?

Crossbar architectures (utilizing technologies like RRAM, PCM, or specialized SRAM) are exceptionally efficient at performing Vector-Matrix Multiplication (VMM) in the analog domain. However, mapping Convolutional Neural Networks (CNNs) to these hardwares is challenging due to massive data reuse.

To maintain high throughput, compilers often "unfold" the network, duplicating weights across many physical crossbars. This leads to a massive "resource explosion." Traditional pruning methods (like element-wise or channel-wise pruning) don't solve this because they don't respect the physical boundaries of the hardware. If a crossbar contains even a few residual non-zero weights, the entire unit—including its power-hungry ADCs, DACs, and peripheral circuits—must remain active. To truly save energy and area, you must be able to remove the entire crossbar unit.

Methods

The authors propose a Crossbar-Aware Pruning framework that treats hardware constraints as first-class citizens during the optimization process.

  1. $L_0$-norm Constrained Optimization: Instead of using $L_1$ or $L_2$ regularization (which encourages small weights), they use an $L_0$ constraint to strictly limit the number of non-zero hardware units.
  2. LGD with RPP: To solve the NP-hard $L_0$ problem, they develop $L_0$-norm constrained Gradient Descent (LGD) coupled with Relaxant Probabilistic Projection (RPP). RPP is a clever alternative to deterministic pruning; instead of just cutting the smallest weights, it uses a probabilistic approach to select candidates, which helps the model converge to a better sparse solution.
  3. Sparsity Grains:
    • Crossbar-grain: Pruning entire crossbar units.
  4. Column-grain with Recombination: Pruning specific columns within crossbars and then "recombining" the remaining non-zero columns from different units into a single, densely packed crossbar.
  5. Input Feature Map (FM) Reordering: They propose a strategy to reorder input channels based on their "importance" (contribution to the output). By grouping important channels together, they make it easier for the LGD solver to prune away entire groups of less-important channels.

Datasets

The framework was evaluated on standard computer vision benchmarks:

  • CIFAR-10: Using a VGG8 model.
  • ImageNet: Using larger-scale VGG16 and ResNet18 models to demonstrate scalability.

Key Insights and Ideas

  • Hardware-Software Co-design: The "aha moment" here is that pruning for hardware isn't about making the weight matrix sparse; it's about making the hardware utilization sparse.
  • The Power of Recombination: Column-grain pruning is significantly more accurate than crossbar-grain pruning because it is more fine-grained. However, it only saves hardware if you have a mechanism to physically "re-pack" those columns into fewer crossbars.
  • Probabilistic Pruning: By using RPP, the authors avoid the "dictatorial" nature of hard-threshold pruning. This allows the optimization process to "explore" which hardware units are truly redundant, leading to higher final accuracy.
  • Layer Sensitivity: The authors found that front-end layers benefit immensely from input reordering, while deeper layers are more resilient to pruning but less affected by reordering, likely because deep features are more abstract and "equally important."

Key Results

The method achieves significant hardware savings across different architectures:

  • VGG16 (ImageNet): Reduced compute crossbars by 59.8% with a ~4% accuracy drop (before fine-tuning).
  • ResNet18 (ImageNet): Reduced compute crossbars by 44.2%. The authors noted ResNet is more sensitive to pruning due to its already compact structure.
  • VGG8 (CIFAR-10): Achieved a massive 72.1% reduction in compute crossbars.
  • Energy/Area: By removing the peripheral circuits associated with pruned crossbars, the framework provides a direct path to reducing the physical footprint and power leakage of AI chips.

Limitations and Future Work

  • Non-Compute Overhead: The authors acknowledge that while they successfully prune "compute" crossbars, many crossbars used for data buffering or activation reduction remain untouched. Future work is needed to address these "auxiliary" resources.
  • Deep Layer Optimization: The current input reordering strategy loses effectiveness in the deepest layers of the network. A more sophisticated reordering or grouping method might be required for those stages.
  • Device Non-idealities: While the paper briefly touches on device variation and quantization (showing that <5-bit quantization causes significant issues), more work is needed to integrate real-world memristor defects (like stuck-at-faults) directly into the pruning optimization loop.

Summary updated

Abstract

Crossbar architecture has been widely adopted in neural network accelerators due to the efficient implementations on vector-matrix multiplication operations. However, in the case of convolutional neural networks (CNNs), the efficiency is compromised dramatically because of the large amounts of data reuse. Although some mapping methods have been designed to achieve a balance between the execution throughput and resource overhead, the resource consumption cost is still huge while maintaining the throughput. Network pruning is a promising and widely studied method to shrink the model size, whereas prior work for CNNs compression rarely considered the crossbar architecture and the corresponding mapping method and cannot be directly utilized by crossbar-based neural network accelerators. This paper proposes a crossbar-aware pruning framework based on a formulated <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{0}$ </tex-math></inline-formula> -norm constrained optimization problem. Specifically, we design an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L_{0}$ </tex-math></inline-formula> -norm constrained gradient descent with relaxant probabilistic projection to solve this problem. Two types of sparsity are successfully achieved: 1) intuitive crossbar-grain sparsity and 2) column-grain sparsity with output recombination, based on which we further propose an input feature maps reorder method to improve the model accuracy. We evaluate our crossbar-aware pruning framework on the median-scale CIFAR10 data set and the large-scale ImageNet data set with VGG and ResNet models. Our method is able to reduce the crossbar overhead by 44%–72% with insignificant accuracy degradation. This paper significantly reduce the resource overhead and the related energy cost and provides a new co-design solution for mapping CNNs onto various crossbar devices with much better efficiency.
BibTeX
@inproceedings{liang2018crossbar,
  author = {Ling Liang and Lei Deng and Yueling Zeng and Xing Hu and Yu Ji and Xin Ma and Guoqi Li and Yuan Xie},
  title = {{Crossbar-Aware Neural Network Pruning}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=TiGfE7zjSdg},
}

Loading related content...

Figure