Compressing Deep Networks by Neuron Agglomerative Clustering

Li-Na Wang, Wenxue Liu, Xiang Liu, Guoqiang Zhong, Partha Pratim Roy, Junyu Dong, Kaizhu Huang

TL;DR

This paper introduces Neuron Agglomerative Clustering (NAC), a novel method for compressing deep neural networks by merging similar neurons and their connections. NAC significantly reduces model size and computational cost without requiring additional libraries or hardware, achieving comparable or even improved accuracy on benchmark datasets.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved remarkable success in various fields, but their complexity often leads to massive storage requirements and long inference times. This makes it challenging to deploy these powerful models on resource-constrained devices like mobile phones. While existing compression techniques like pruning and quantization can reduce model size, they often require specialized libraries, hardware, or complex manual tuning. The authors aim to address this by developing a more straightforward and effective method for compressing DNNs that directly tackles the redundancy within neurons and their connections.

Methods

The core of this paper is the Neuron Agglomerative Clustering (NAC) method. The approach works by:

  • Representing Neurons: Each neuron in a layer is represented by its weights and bias.
  • Agglomerative Clustering: For each layer, an agglomerative clustering algorithm is applied to group similar neurons based on their Euclidean distance. This is a "bottom-up" approach where individual neurons are initially treated as clusters, and then the closest clusters are merged iteratively.
  • Neuron Agglomeration: Similar neurons within a cluster are then merged into a single "cluster centroid." This centroid effectively replaces the group of similar neurons.
  • Connection Agglomeration: Crucially, the connections to and from these agglomerated neurons are also merged and reconnected to the new cluster centroid. This ensures that the network's structure is maintained while reducing the number of parameters.
  • Fine-tuning: After the compression, the network is fine-tuned to recover any potential accuracy loss and further optimize performance.

A key advantage highlighted is that NAC does not create sparse connections, thus avoiding the need for specialized sparse matrix libraries or hardware. It also operates on weights and biases directly, without requiring the original training data during the compression phase.

Datasets

The paper evaluates the proposed NAC method on the following standard datasets:

  • MNIST: A dataset for handwritten digit recognition, consisting of 70,000 grayscale images (28x28 pixels).
  • CIFAR-10: A dataset of 60,000 color images (32x32 pixels) across 10 classes.
  • CIFAR-100: A dataset of 60,000 color images (32x32 pixels) across 100 classes.

Key Insights and Ideas

  • Neuron Redundancy is Pervasive: The paper visually demonstrates that even in well-trained deep neural networks, many neurons within the same layer learn very similar features, indicating significant redundancy.
  • Agglomerative Clustering is a Natural Fit: The bottom-up nature of agglomerative clustering is well-suited for identifying and merging these similar neurons, leading to a more stable and effective compression compared to methods like k-means which can suffer from random initialization issues.
  • Compressing Connections is Crucial: Simply merging neurons without appropriately merging their associated connections would break the network. The paper emphasizes the importance of agglomerating these connections to maintain network functionality.
  • No Need for Sparse Libraries: NAC's ability to reduce model size without introducing sparsity is a significant practical advantage, simplifying deployment on standard hardware.
  • Fine-tuning Recovers and Enhances Performance: The fine-tuning step after compression is shown to not only recover any lost accuracy but in some cases, even surpass the original network's performance.

Key Results

  • Significant Parameter Reduction: On VGGNet-16 for CIFAR-10, NAC achieved a parameter reduction of 92.96% with minimal accuracy loss. For CIFAR-100, it achieved 81.10% parameter reduction.
  • Improved Accuracy: In several experiments, the compressed network with fine-tuning achieved higher accuracy than the original network. For example, on CIFAR-100, Model-B (using agglomerative clustering) achieved a +0.30% accuracy improvement over the baseline VGGNet.
  • Effectiveness Across Architectures: NAC demonstrated effectiveness on both Deep Belief Networks (DBNs) and Convolutional Neural Networks (CNNs), including the complex VGGNet-16.
  • Superiority over K-Means: When compared to k-means clustering for neuron merging on CIFAR datasets, NAC consistently yielded better results, highlighting the robustness of agglomerative clustering.
  • Competitive with State-of-the-Art: NAC was compared to other network compression methods like Network Slimming and Pruning Filters, showing comparable or superior accuracy gains at similar compression ratios.

Limitations and Future Work

  • Computational Cost of Clustering: While NAC avoids sparse matrix operations, the agglomerative clustering itself can be computationally intensive, especially for very large layers. The paper mentions that the process is performed layer-wise, which helps manage this.
  • Potential for Further Compression: The paper suggests that NAC can be easily combined with other compression techniques like quantization and pruning to achieve even more compact networks. This indicates a clear avenue for future research.
  • Generalizability to Other Architectures: While tested on DBNs, CNNs, and VGGNet, further exploration on a wider range of modern deep learning architectures (e.g., Transformers) could be beneficial.

Summary updated

BibTeX
@inproceedings{wang2020compressing,
  author = {Li-Na Wang and Wenxue Liu and Xiang Liu and Guoqiang Zhong and Partha Pratim Roy and Junyu Dong and Kaizhu Huang},
  title = {{Compressing Deep Networks by Neuron Agglomerative Clustering}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=6KE9UjZl_h},
}
Figure