EvoPruneDeepTL: An evolutionary pruning model for transfer learning based deep neural networks

Javier Poyatos, Daniel Molina, Aritz D. Martinez, Javier Del Ser, Francisco Herrera

TL;DR

This paper introduces EvoPruneDeepTL, a novel approach that uses evolutionary algorithms to prune the final layers of deep neural networks trained with transfer learning. By optimizing for accuracy and efficiency simultaneously, EvoPruneDeepTL can outperform traditional pruning methods and even achieve better results through feature selection.

What Problem Does This Paper Address?

Deep learning models achieve impressive results but often require massive datasets, which are not always available. Transfer learning offers a solution by leveraging pre-trained models, but optimizing the final layers for a new task remains a computationally intensive challenge. Traditional pruning methods aim to reduce model complexity but often at the cost of performance. This paper addresses the need for a method that can optimize the final layers of transfer-learned models to improve performance and reduce complexity, rather than just minimize accuracy loss.

Methods

The core of EvoPruneDeepTL is a genetic algorithm (GA) that evolves sparse representations of the fully-connected layers in a transfer-learned deep neural network. Instead of training all connections, the GA learns to identify and keep only the most crucial ones. The paper explores two main encoding strategies for the GA:

  • Neuron Encoding: Each gene in the chromosome represents a neuron, determining if it's active or inactive.
  • Connection Encoding: Each gene represents a specific connection between neurons, allowing for finer-grained pruning.

EvoPruneDeepTL can be applied to prune either single or multiple fully-connected layers. A key innovation is that by focusing the evolutionary search on the features extracted by the pre-trained network (when using a single fully-connected layer), the method effectively performs feature selection, identifying the most relevant input features for the task.

Datasets

The authors evaluated EvoPruneDeepTL on six diverse datasets:

  • SRSMAS: Coral reef types classification.
  • RPS: Rock Paper Scissors game gesture identification.
  • LEAVES: Citrus leaf health classification.
  • PAINTING: Painting type recognition.
  • CATARACT: Medical domain dataset for eye disease classification.
  • PLANTS: Plant species identification.

These datasets were chosen for their suitability for transfer learning, requiring less training time and inference.

Key Insights and Ideas

  • Pruning for Improvement, Not Just Reduction: The paper demonstrates that pruning can be used not only to reduce model size but also to actively improve accuracy by identifying and retaining the most impactful connections or features.
  • Feature Selection as a Powerful Pruning Strategy: When applied to the extracted features, EvoPruneDeepTL's evolutionary approach effectively acts as feature selection, often yielding superior results compared to pruning the network's connections directly. This highlights the importance of selecting the right input representations.
  • Neuron-wise Pruning Outperforms Connection-wise: The study found that evolving neurons (neuron encoding) generally led to more robust results and better model reduction compared to evolving individual connections.
  • Adaptability to Different Feature Extractors: EvoPruneDeepTL shows strong generalization capabilities, performing well with various pre-trained feature extractors like ResNet-50, DenseNet-121, and VGG-19.
  • Robustness to Class Imbalance and Changes: The method demonstrates resilience to class imbalances and can adapt to evolving datasets where class importance changes.

Key Results

  • Accuracy Improvements: EvoPruneDeepTL consistently achieved higher accuracy than baseline models (not pruned, best fixed) across all tested datasets, with improvements ranging from a few percentage points to significant gains (e.g., 4.7% on RPS for two layers).
  • Significant Compression: The method achieved substantial reductions in the number of active neurons/connections. For instance, on SRSMAS with one layer, active neurons were reduced from 100% to 25%.
  • Feature Selection Superiority: In most cases, the feature selection variant of EvoPruneDeepTL outperformed both direct pruning methods and traditional pruning techniques from the literature. For example, on RPS with two layers, feature selection achieved 0.985 accuracy compared to 0.978 for two-layer pruning.
  • Comparison to State-of-the-Art Pruning: EvoPruneDeepTL generally outperformed established pruning methods like Polynomial Decay, Weight Pruning, and Neuron Pruning in terms of accuracy, although it was slower in execution time.
  • Robustness Confirmed: Centered Kernel Alignment (CKA) analysis showed that EvoPruneDeepTL's results are not due to randomness and that the model effectively learns essential neurons.

Limitations and Future Work

  • Execution Time: The primary limitation acknowledged is the computational cost associated with evolutionary algorithms. While EvoPruneDeepTL achieves better performance, it is slower than traditional pruning methods. The authors suggest that this is acceptable in domains like medical diagnosis where accuracy is paramount.
  • Future Work: The authors propose several avenues for future research:
  • Extending the evolutionary search to higher layers of the neural hierarchy.
  • Incorporating advanced evolutionary computation concepts (e.g., niching, co-evolution).
  • Reformulating the pruning problem to include the selection of activation functions for neurons.

Summary updated

Abstract

In recent years, Deep Learning models have shown a great performance in complex optimization problems. They generally require large training datasets, which is a limitation in most practical cases. Transfer learning allows importing the first layers of a pre-trained architecture and connecting them to fully-connected layers to adapt them to a new problem. Consequently, the configuration of the these layers becomes crucial for the performance of the model. Unfortunately, the optimization of these models is usually a computationally demanding task. One strategy to optimize Deep Learning models is the pruning scheme. Pruning methods are focused on reducing the complexity of the network, assuming an expected performance penalty of the model once pruned. However, the pruning could potentially be used to improve the performance, using an optimization algorithm to identify and eventually remove unnecessary connections among neurons. This work proposes EvoPruneDeepTL, an evolutionary pruning model for Transfer Learning based Deep Neural Networks which replaces the last fully-connected layers with sparse layers optimized by a genetic algorithm. Depending on its solution encoding strategy, our proposed model can either perform optimized pruning or feature selection over the densely connected part of the neural network. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show the contribution of EvoPruneDeepTL and feature selection to the overall computational efficiency of the network as a result of the optimization process. In particular, the accuracy is improved, reducing at the same time the number of active neurons in the final layers.
BibTeX
@inproceedings{poyatos2023evoprunedeeptl,
  author = {Javier Poyatos and Daniel Molina and Aritz D. Martinez and Javier Del Ser and Francisco Herrera},
  title = {{EvoPruneDeepTL: An evolutionary pruning model for transfer learning based deep neural networks}},
  booktitle = {Neural Networks},
  year = {2023},
  url = {https://openreview.net/forum?id=eS2XHodfaK},
}
Figure