Deep Compression: Compressing Deep Neural Networks with Pruning

Trained Quantization and Huffman Coding Song Han, Huizi Mao, Bill Dally

Introduction

"Deep Compression" introduces a three-stage pipeline—pruning, trained quantization, and Huffman coding—that reduces the storage requirements of deep neural networks by 35x to 49x without sacrificing accuracy. By shrinking models like VGG-16 from 552MB to just 11.3MB, this technique allows complex AI to run entirely within high-speed on-chip cache. This drastically improves energy efficiency and enables the deployment of sophisticated models directly on mobile devices.

The Problem: Why Compress?

Deep neural networks (DNNs) have become the gold standard for computer vision, but they are notoriously "heavy." Modern architectures like AlexNet and VGG-16 require hundreds of megabytes of storage and massive memory bandwidth.

This creates two major bottlenecks:

  1. Deployment Constraints: Mobile app stores often have strict limits on binary sizes (e.g., the 100MB Wi-Fi download limit), making it difficult to ship "smart" applications without forcing users to download massive auxiliary files.
  2. Energy Consumption: In mobile and embedded systems, the most energy-intensive operation isn't the math—it's moving data. Accessing off-chip DRAM memory consumes orders of magnitude more energy than accessing on-chip SRAM cache or performing arithmetic operations. To make AI "green" and mobile-friendly, models must fit into tiny, fast, on-chip memory.

The Deep Compression Pipeline

The authors propose a sequential, three-step "Deep Compression" pipeline to tackle these bottlenecks.

The three-stage Deep Compression pipeline: Pruning, Quantization, and Huffman Encoding.

1. Pruning

First, the network is trained to identify which connections are actually important. Connections with weights below a certain threshold are removed, and the network is retrained to "heal" the remaining sparse connections. This step alone reduces the number of parameters by 9x to 13x.

2. Trained Quantization (Weight Sharing)

Instead of every connection having a unique 32-bit floating-point value, the authors force multiple connections to share the same weight. They use k-means clustering to group weights and store only a small index (e.g., 5 bits) pointing to a "codebook" of shared values.

Weight sharing and fine-tuning via k-means clustering. Gradients are grouped by cluster index and reduced to update the shared centroids.

Crucially, they fine-tune these shared values during retraining. The gradients for all weights in a cluster are summed together to update the shared centroid, ensuring the quantization process does not degrade the network's accuracy.

3. Huffman Coding

Finally, they apply Huffman coding—a classic lossless compression technique—to the quantized weights and sparse indices. This takes advantage of the fact that some weight indices appear much more frequently than others, squeezing out the remaining redundancy and saving an additional 20% to 30% of storage.

Key Insights

  • The Synergy of Pruning and Quantization: A major insight from the paper is that pruning and quantization do not interfere with each other. You can prune a network heavily and then quantize it, and the accuracy remains stable. In fact, quantization works better on a pruned network because there are fewer weights to represent, making the clustering process more effective.
  • Linear Initialization is King: When choosing the initial shared weights for quantization, the authors found that "Linear Initialization" (spacing centroids equally across the range of values) outperformed more complex density-based methods. This is because it preserves the rare, large-magnitude weights that are most critical to the network's performance.
  • SRAM vs. DRAM: The ultimate goal of this pipeline is to get the model size small enough to fit into on-chip SRAM. The paper highlights that a DRAM access costs roughly 640pJ, while an SRAM access costs only 5pJ. By fitting the model in cache, they effectively eliminate the "memory wall."

Key Results

The researchers evaluated their method on standard benchmarks, including MNIST (LeNet architectures) and ImageNet (AlexNet and VGG-16). The results are striking because they achieve massive compression with zero loss in accuracy:

  • AlexNet: Compressed by 35x (from 240MB down to 6.9MB).
  • VGG-16: Compressed by 49x (from 552MB down to 11.3MB).
  • Efficiency: On mobile GPUs, the compressed networks provided a 3x to 4x speedup and were 3x to 7x more energy-efficient than their uncompressed counterparts.
  • Bit Reduction: They successfully reduced the bits per connection from 32 bits down to just 3–5 bits for most layers.

Limitations and Future Work

While the compression results are excellent, the authors note that standard software libraries (like BLAS) are designed for "dense" math and don't natively support the "sparse and quantized" format produced by this pipeline. To see the full speed and energy benefits, one needs:

  • Custom Kernels: Software specifically written to handle indirect lookups and relative indexing.
  • Specialized Hardware: The authors suggest that custom ASIC architectures are the next step to fully exploit these compressed models (which they later explored with the "EIE" accelerator).
  • Layer-Specific Sensitivity: The paper notes that convolutional layers are more sensitive to quantization than fully connected layers, suggesting that future work could further optimize the bit-allocation per layer.

Summary updated

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.
BibTeX
@inproceedings{han2016deep,
  author = {Trained Quantization and Huffman Coding Song Han and Huizi Mao and Bill Dally},
  title = {{Deep Compression: Compressing Deep Neural Networks with Pruning}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1510.00149},
}
Figure