Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Song Han, Huizi Mao, William J. Dally

TL;DR

This paper introduces "Deep Compression," a three-stage pipeline (pruning, trained quantization, and Huffman coding) that dramatically reduces the storage and computational requirements of deep neural networks by up to 49x without sacrificing accuracy. This breakthrough enables the deployment of complex neural networks on resource-constrained embedded and mobile devices, offering significant speedups and energy efficiency improvements.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved state-of-the-art performance in many domains, particularly computer vision. However, their massive size and computational demands make them impractical for deployment on embedded systems and mobile devices, which have limited storage, memory bandwidth, and processing power. This limitation hinders the widespread adoption of advanced AI in everyday applications. For instance, large models like AlexNet (240MB) and VGG-16 (552MB) are too big for typical mobile app size restrictions, and their high memory access requirements lead to substantial energy consumption, quickly draining batteries on mobile devices.

Methods

The authors propose a three-stage compression pipeline called "Deep Compression":

  1. Pruning: This initial step identifies and removes redundant connections (weights) in the neural network. The process involves training the network, pruning connections with weights below a certain threshold, and then retraining the network to fine-tune the remaining connections. This stage reduces the number of connections by 9x to 13x.
  2. Trained Quantization and Weight Sharing: Following pruning, weights are quantized to reduce the number of bits needed to represent them. This is achieved by grouping similar weights into clusters, where all weights within a cluster share a single "centroid" value. The network is then retrained to fine-tune these shared centroid values. This stage reduces the bits per connection from 32 to as few as 5, enabling significant storage reduction. The paper explores different initialization methods for these centroids, finding that linear initialization performs best.
  3. Huffman Coding: The final stage applies Huffman coding to the quantized weights and indices. This technique assigns shorter codes to more frequent symbols (weights or indices) and longer codes to less frequent ones, further compressing the data by exploiting the biased distribution of the quantized values.

The key insight is that these three stages are largely independent and can be applied sequentially, leading to a multiplicative compression effect without compromising accuracy.

Datasets

The authors evaluate their method on the following datasets:

  • MNIST: Used for experiments with LeNet-300-100 and LeNet-5 networks.
  • ImageNet (ILSVRC-2012): Used for evaluating the compression of larger networks like AlexNet and VGG-16.

The paper does not introduce new benchmarks but rather applies its compression techniques to established, large-scale datasets to demonstrate their effectiveness on real-world models.

Key Insights and Ideas

  • Synergy of Compression Techniques: The most significant insight is that pruning, quantization, and Huffman coding work exceptionally well together. Applying them sequentially leads to a much greater compression ratio than any single method alone, without a significant drop in accuracy.
  • Pruning Enables Better Quantization: Pruning reduces the number of weights, making the subsequent quantization step more effective. With fewer weights to represent, the quantization process can achieve higher accuracy with fewer bits.
  • Fitting Models into On-Chip Memory: The extreme compression achieved allows entire neural networks to fit into fast, low-power on-chip SRAM, eliminating the need for slower, power-hungry off-chip DRAM. This is crucial for real-time inference on embedded systems.
  • Linear Initialization for Centroids: For the quantization stage, linear initialization of weight centroids proved to be superior to random or density-based methods, leading to better accuracy. This is because it better preserves the distribution of large weights, which are important for network performance.

Key Results

  • Storage Reduction:
    • AlexNet: Reduced from 240MB to 6.9MB (35x compression).
  • VGG-16: Reduced from 552MB to 11.3MB (49x compression).
  • LeNet-5: Compressed by 39x.
  • LeNet-300-100: Compressed by 40x.
  • Accuracy: No loss of accuracy was observed for AlexNet and VGG-16 on ImageNet. For LeNet models on MNIST, accuracy remained comparable.
  • Speedup: Compressed networks achieved 3x to 4x layerwise speedup on average across CPU, GPU, and mobile GPU.
  • Energy Efficiency: Compressed networks consumed 3x to 7x less energy on average.
  • Quantization Bits: Achieved 8-bit quantization for CONV layers and 5-bit for FC layers without accuracy loss. Aggressive 4/2 bit quantization resulted in a small accuracy drop (around 2%).

Limitations and Future Work

  • Hardware Support for Quantized Networks: The paper notes that current off-the-shelf libraries (cuSPARSE, MKL SPBLAS) do not fully support the sparse and quantized network structures, limiting the full realization of the speedup and energy benefits.
  • Future Work: The authors suggest developing custom GPU kernels or hardware (ASICs) to fully leverage the compressed network structures. This would enable specialized architectures that can efficiently traverse sparse and quantized networks, further optimizing for on-chip SRAM access and reducing energy consumption. They also mention that the full advantage of Deep Compression, especially fitting the model into cache, is not yet fully unveiled due to current library limitations.

Summary updated

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.
BibTeX
@inproceedings{han2016deep,
  author = {Song Han and Huizi Mao and William J. Dally},
  title = {{Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1510.00149},
}
Figure