Shannon Strikes Again! Entropy-based Pruning in Deep Neural Networks for Transfer Learning under Extreme Memory and Computation Budgets

Gabriele Spadaro, Riccardo Renzulli, Andrea Bragagnolo, Jhony H. Giraldo, Attilio Fiandrotti, Marco Grangetto, Enzo Tartaglione

TL;DR

This paper introduces ENTROPI, a novel algorithm that prunes deep neural networks based on activation entropy to improve transfer learning performance, especially for resource-constrained devices (TinyML). They found that models with higher activation entropy on the target task generalize better, leading to significant accuracy gains in low-memory scenarios.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is computationally expensive and requires vast amounts of data, making them unsuitable for edge devices with limited memory and processing power (TinyML). While pruning can reduce model size, it doesn't always guarantee improved performance in transfer learning, where a model trained on one task is adapted to another. This paper tackles the challenge of achieving effective transfer learning under extreme memory and computation constraints, aiming to make powerful DNNs usable on resource-scarce platforms.

Methods

The core of this work is the ENTROPI algorithm, which combines iterative magnitude pruning with entropy-based model selection for transfer learning. Here's a breakdown:

  • Iterative Magnitude Pruning: The algorithm starts by training a model on a source task and then iteratively prunes it. Pruning involves removing weights based on their magnitude, controlled by a parameter 'α' (representing the fraction of remaining parameters). This process is repeated with decreasing values of α.
  • Entropy Estimation: For each pruned model, the authors estimate the entropy of its activations on the target task's training data. Entropy, in this context, measures the uncertainty or randomness of the activation distribution.
  • Entropy-Based Selection: The key insight is that the pruned model exhibiting the highest activation entropy on the target task is selected as the best candidate for transfer learning. This pruned model's encoder is then used.
  • Fine-tuning: Finally, the classifier head of the selected pruned model is trained on the target task, while the encoder is kept frozen.

This approach differs from standard pruning by using activation entropy on the target task as a criterion for selecting the best pruned model, rather than just focusing on sparsity or performance on the source task.

Datasets

The paper uses the following datasets:

  • Source Task Training: CIFAR-10 (for training and pruning the initial models).
  • Target Task Evaluation: CIFAR-100, VLCS, and PACS.

Key Insights and Ideas

  • Entropy as a Proxy for Generalization: The most significant finding is that higher activation entropy in the pruned model's encoder, when evaluated on the target task, correlates with better generalization performance on that target task. This suggests that a more "uncertain" or diverse representation space might be more adaptable.
  • The "Sweet Spot" of Pruning: The authors observe that performance on the target task peaks at a specific pruning level (low α values, around 0.01), which also corresponds to the peak in activation entropy. Beyond this point, excessive pruning leads to performance degradation.
  • Specialization vs. Forgetting Regions: The paper identifies two conceptual regions during pruning: a "specialization region" where the encoder aids classification by projecting features into a lower-dimensional space, and a "forgetting region" where complexity hinders source task information extraction. The optimal pruning point seems to lie between these.
  • TinyML Viability: The ENTROPI approach is shown to be highly effective for extremely low memory budgets (tens of kB for the classifier head), making it suitable for TinyML applications where full fine-tuning is impossible.

Key Results

  • Accuracy Improvements: Across all tested datasets (CIFAR-100, VLCS, PACS) and architectures (ResNet18, MobileNet-v3), ENTROPI consistently outperformed the dense (unpruned) models in transfer learning. Top-1 accuracy gains of up to 8% were observed.
  • Low Pruning Ratios: The optimal pruning ratios (α values) identified were very low, often around 0.01, indicating that highly sparse models can achieve superior transfer learning performance.
  • Memory Efficiency: The paper highlights that by only fine-tuning the classifier head, the memory footprint is drastically reduced (tens of kB) compared to full fine-tuning (hundreds of MB), making it practical for embedded systems.

Limitations and Future Work

  • Computational Cost of Pruning: While the final inference is efficient, the iterative pruning process itself can be computationally intensive.
  • Generalizability of Entropy Metric: The paper focuses on activation entropy. Future work could explore other information-theoretic measures or different pruning criteria.
  • Dataset Specificity: While tested on multiple datasets, further validation on a wider range of tasks and domains would be beneficial.
  • TinyML Context: The authors emphasize the suitability for extreme contexts like embedded AI and TinyML, suggesting that this approach paves the way for adapting large architectures directly on IoT devices.

Summary updated

Abstract

Deep neural networks have become the de-facto standard across various computer science domains. Nonetheless, effectively training these deep networks remains challenging and resource-intensive. This paper investigates the efficacy of pruned deep learning models in transfer learning scenarios under extremely low memory budgets, tailored for TinyML models. Our study reveals that the source task’s model with the highest activation entropy outperforms others in the target task. Motivated by this, we propose an entropy-based Efficient Neural Transfer with Reduced Overhead via PrunIng (ENTROPI) algorithm. Through comprehensive experiments on diverse models (ResNet18 and MobileNet-v3) and target datasets (CIFAR-100, VLCS, and PACS), we substantiate the superior generalization achieved by transfer learning from the entropy-pruned model. Quantitative measures for entropy provide valuable insights into the reasons behind the observed performance improvements. The results underscore ENTROPI’s potential as an efficient solution for enhancing generalization in data-limited transfer learning tasks.
BibTeX
@inproceedings{spadaro2023shannon,
  author = {Gabriele Spadaro and Riccardo Renzulli and Andrea Bragagnolo and Jhony H. Giraldo and Attilio Fiandrotti and Marco Grangetto and Enzo Tartaglione},
  title = {{Shannon Strikes Again! Entropy-based Pruning in Deep Neural Networks for Transfer Learning under Extreme Memory and Computation Budgets}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00165},
}
Figure