Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks

Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William M. Leiserson, Sage Moore, Nir Shavit, Dan AlistarhICML 2020 · OpenReview

TL;DR

While most model compression focuses on pruning weights, this paper shifts the spotlight to activation sparsity—the zeros produced by ReLU functions. By combining a novel "Forced Activation-Threshold" (FATReLU) and Square Hoyer regularization with a custom-built sparse CPU kernel, the authors achieve up to 2.57x inference speedups on standard ImageNet models with negligible accuracy loss.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are computationally heavy. To make them run faster, researchers usually "prune" the weights (set small connections to zero) or "quantize" them (use fewer bits). However, there is a third source of efficiency: activation maps.

Because of the Rectified Linear Unit (ReLU), which turns all negative values into zeros, many activations in a network are naturally zero. Despite this, standard deep learning libraries (like PyTorch or TensorFlow) treat these maps as "dense," performing math on those zeros anyway. The authors identify two gaps:

  1. Induction Gap: We don't have great ways to increase the number of zeros in activations without crashing model accuracy.
  2. Exploitation Gap: Even if a model is 90% sparse, standard CPUs and GPUs don't run it any faster because software support for sparse-input convolutions is lacking.

Methods

The authors propose a two-pronged approach: a training strategy to create more zeros and a software engine to skip them.

1. Inducing Sparsity

  • Square Hoyer Regularization: Instead of the common L1 regularization, the authors use the "Square Hoyer" metric. It is scale-invariant and differentiable, making it more effective at "pushing" activations toward zero during fine-tuning.
  • FATReLU (Forced Activation-Threshold ReLU): The authors introduce a modified activation function that explicitly zeros out any value below a learned threshold $T$.
  • Dynamic Thresholding: They use a binary search algorithm to find the maximum possible threshold for each layer that stays within a tiny accuracy loss budget (e.g., 0.2%).

The impact of Hoyer regularization is visually striking when compared to standard L1 regularization, as it successfully forces a large portion of activations to zero while maintaining a healthy distribution of necessary non-zero values.

Histograms comparing activation distributions across baseline, L1, and Hoyer regularization methods.

2. Exploiting Sparsity

  • Sparse-Input Convolution Kernel: The authors wrote a custom C++ inference engine. It represents sparse activations using a Compressed Sparse Row (CSR) format.
  • AVX-512 Optimization: To avoid the "branching" problem (where the CPU slows down trying to decide whether to skip a zero), they use specialized Intel instructions like vcompress and vfmadd. This allows the CPU to pack non-zero values together and perform vector math at full speed, effectively "ignoring" the zeros without a performance penalty.

Datasets

The primary evaluation was performed on the ImageNet (ILSVRC2012) dataset, the gold standard for large-scale image classification. They also validated the general trends on CIFAR-10 and CIFAR-100.

The methods were tested across several popular architectures:

  • ResNet (18, 50, and 101)
  • MobileNetV1
  • Wide ResNet variants

Key Insights and Ideas

  • Hoyer > L1: The paper demonstrates that Hoyer regularization is significantly more powerful than L1 for activations. It can double the sparsity gains compared to previous L1-based methods while requiring much less retraining time (10 epochs vs. 90).
  • The "Sharp" Threshold: The authors discovered a "bi-modal" behavior in activation sensitivity. You can increase the threshold (and thus sparsity) quite far with almost zero impact on accuracy, but once you hit a specific "sharp" limit, accuracy collapses. This makes the search for optimal thresholds much simpler.
  • Depth and Width Breed Sparsity: Naturally, deeper and wider networks exhibit higher activation sparsity. "Higher-level" features in deeper layers tend to be more specialized and thus more sparse.
  • Software is the Key: The most important "aha" moment is that activation sparsity is only a theoretical win until you have a kernel that can handle sparse inputs at the register level. Their AVX-512 implementation proves that commodity CPUs can be competitive with GPUs when sparsity is leveraged correctly.

Key Results

The authors compared their "Boosted Hoyer" method against optimized dense baselines (like Intel MKL-DNN). They successfully increased activation sparsity from a "natural" ~50% to 65%–84% across various models.

These sparsity gains translated directly into massive layer-wise and end-to-end speedups, particularly in the deeper layers of the network where sparsity is highest.

Layer-by-layer inference time comparison showing significant speedups for Hoyer-regularized models over the dense baseline.

Overall end-to-end speedups included:

  • ResNet50: Achieved a 1.75x speedup.
  • Wide ResNet101: Achieved a massive 2.57x speedup.
  • MobileNet: Achieved a 1.78x speedup.

Crucially, all speedups were achieved with < 0.3% loss in top-1 accuracy compared to the original dense models.

Limitations and Future Work

  • Hardware Specificity: The current high-performance kernels are tailored for Intel CPUs with AVX-512 support. While the theory applies to GPUs, the implementation would look very different due to how GPU warps handle branching.
  • Fine-tuning Required: While the "static" version of their method requires no retraining, the best results (the 2x+ speedups) require a short fine-tuning phase.
  • Memory-Bound Layers: Some layers are limited by memory bandwidth rather than raw computation; the authors plan to explore specialized strategies for these specific bottlenecks in future work.
  • Combining Techniques: The paper does not yet combine activation sparsity with weight pruning or quantization, which could potentially lead to even more dramatic cumulative gains.

Summary updated

Abstract

Optimizing convolutional neural networks for fast inference has recently become an extremely active area of research. One of the go-to solutions in this context is weight pruning, which aims to red...
BibTeX
@inproceedings{kurtz2020inducing,
  author = {Mark Kurtz and Justin Kopinsky and Rati Gelashvili and Alexander Matveev and John Carr and Michael Goin and William M. Leiserson and Sage Moore and Nir Shavit and Dan Alistarh},
  title = {{Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=yRhob_jcFoU},
}
Figure