Towards Evolutional Compression

Yunhe Wang, Chang Xu, Jiayan Qiu, Chao Xu, Dacheng TaoarXiv 2017 · OpenReview

Abstract: Compressing convolutional neural networks (CNNs) is essential for transferring the success of CNNs to a wide variety of applications to mobile devices. In contrast to directly recognizing subtle weights or filters as redundant in a given CNN, this paper presents an evolutionary method to automatically eliminate redundant convolution filters. We represent each compressed network as a binary individual of specific fitness. Then, the population is upgraded at each evolutionary iteration using genetic operations. As a result, an extremely compact CNN is generated using the fittest individual. In this approach, either large or small convolution filters can be redundant, and filters in the compressed network are more distinct. In addition, since the number of filters in each convolutional layer is reduced, the number of filter channels and the size of feature maps are also decreased, naturally improving both the compression and speed-up ratios. Experiments on benchmark deep CNN models suggest the superiority of the proposed algorithm over the state-of-the-art compression methods.

Introduction

Deep Convolutional Neural Networks (CNNs) have revolutionized computer vision, but they come with a heavy price tag: massive memory requirements and high computational costs. This makes deploying state-of-the-art models on mobile devices a significant challenge. While many researchers have tried to "prune" these networks by cutting out unimportant parts, most rely on simple heuristics—like assuming small weights are useless.

Researchers from Peking University and the University of Sydney introduce Evolutional Compression (ECS), a method that uses Genetic Algorithms to automatically identify and eliminate redundant filters in CNNs. By treating network architectures as "individuals" in a population and evolving them for maximum fitness, they achieved significant compression and speedups without compromising accuracy.

Evolutionary Compression (ECS)

The authors formulated compression as a binary programming problem, where each "individual" in a population is represented by a binary string (a chromosome). Each bit corresponds to a filter in the original CNN: 1 means keep the filter, 0 means discard it.

Evolution of network fitness and architecture over iterations.

To determine which individuals "survive," a fitness function rewards both low classification error and a high compression ratio. The evolutionary loop consists of three primary operations:

  • Selection: The best-performing architectures are kept and duplicated.
  • Crossover: Two "parent" architectures swap fragments of their binary strings to create "offspring" with potentially better traits.
  • Mutation: Bits are randomly flipped to maintain diversity in the population and discover new architectures.

Evaluating a full CNN on a massive dataset like ImageNet is computationally expensive. To accelerate the process, the authors fine-tuned each candidate network on a small subset (10,000 images) for a brief period to estimate its potential performance.

Experimental Results

The method was evaluated on two primary benchmarks: MNIST (using LeNet) and ImageNet (using AlexNet, VGG-16, and ResNet-50). The results across different architectures were impressive:

  • LeNet (MNIST): Achieved 15.52× compression and 5.76× speedup with zero loss in accuracy.
  • VGG-16 (ImageNet): Achieved an 8.81× compression ratio and a 5.88× speedup. The top-5 error only moved from 9.9% to 10.2%, a negligible trade-off for such massive gains.
  • ResNet-50 (ImageNet): Even on this already-optimized architecture, ECS achieved a 4.10× compression and 3.83× speedup.

Key Insights

  • Beyond Small Weights: The evolutionary process often discarded filters with large weights and kept some with small weights, proving that magnitude is a poor proxy for importance.
  • Filter Diversity: After evolution, the remaining filters were more "distinct" from one another. The algorithm naturally moves away from redundancy toward a more efficient, diverse set of feature extractors.
  • Automatic Architecture Search: Unlike manual pruning, this method automatically determines the optimal number of filters for every layer simultaneously, accounting for how changes in one layer affect the next.

Conclusion

While the search time for genetic algorithms remains a consideration, the ECS framework demonstrates that evolutionary strategies can effectively navigate the complex trade-offs of neural network architecture. Future directions include combining this evolutionary approach with other techniques like quantization to achieve even more extreme compression ratios for mobile deployment.

Summary updated

Abstract

Compressing convolutional neural networks (CNNs) is essential for transferring the success of CNNs to a wide variety of applications to mobile devices. In contrast to directly recognizing subtle weights or filters as redundant in a given CNN, this paper presents an evolutionary method to automatically eliminate redundant convolution filters. We represent each compressed network as a binary individual of specific fitness. Then, the population is upgraded at each evolutionary iteration using genetic operations. As a result, an extremely compact CNN is generated using the fittest individual. In this approach, either large or small convolution filters can be redundant, and filters in the compressed network are more distinct. In addition, since the number of filters in each convolutional layer is reduced, the number of filter channels and the size of feature maps are also decreased, naturally improving both the compression and speed-up ratios. Experiments on benchmark deep CNN models suggest the superiority of the proposed algorithm over the state-of-the-art compression methods.
BibTeX
@misc{wang2017towards,
  author = {Yunhe Wang and Chang Xu and Jiayan Qiu and Chao Xu and Dacheng Tao},
  title = {{Towards Evolutional Compression}},
  year = {2017},
  eprint = {1707.08005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.08005},
}
Figure