Paoding: Supervised Robustness-preserving Data-free Neural Network Pruning

Mark Huasong Meng, Guangdong Bai, Sin Gee Teo, Jin Song Dong

TL;DR

This paper introduces a novel "data-free" neural network pruning method that preserves model robustness, a critical factor for real-world deployments where original training data is unavailable. By treating pruning as a progressive, supervised process guided by robustness metrics and employing stochastic optimization, their method significantly outperforms existing data-free techniques in both accuracy and robustness.

What Problem Does This Paper Address?

Deploying powerful deep learning models on resource-constrained devices like smartphones and IoT devices often requires reducing their size and computational demands. Neural network pruning is a common technique for this, but most existing methods assume the pruned model can be fine-tuned or retrained using the original training data. This assumption is often unrealistic in practice, especially due to data privacy concerns and regulations like GDPR, where data controllers are reluctant to share sensitive training data. This creates a need for "data-free" pruning methods that can reduce model size without access to the original data, while also ensuring the pruned model remains robust against unexpected or adversarial inputs in open-world scenarios.

Methods

The authors propose a supervised, progressive, data-free neural network pruning framework. Instead of aggressive one-shot pruning, they adopt a conservative approach that treats pruning as a gradual process. Key elements include:

  • Progressive Pruning: Pruning is done iteratively, cutting off small sets of units and evaluating their impact at each step to minimize mis-pruning of crucial units.
  • Robustness-Guided Pruning: The pruning process is guided by metrics that quantify the model's robustness. Specifically, they use L1-norm (measuring the overall scale of pruning impact on the model's output) and entropy (measuring the similarity of pruning impact across output units) to ensure the impact is uniformly distributed.
  • Stochastic Optimization with Simulated Annealing: To identify which units to prune, they employ a two-stage approach:
  • Candidate Prioritizing: Units are prioritized based on their "saliency," a measure of their importance.
  • Stochastic Sampling: A simulated annealing algorithm is used to sample candidates, guided by the robustness metrics. This helps avoid getting stuck in local optima and finds units with minimal impact on robustness.
  • Primitive Pruning Operation: When pruning a unit (nominee), they don't simply delete it. Instead, they zero out its connections and adjust the parameters of another unit (delegate) in the same layer to cover its duty, minimizing the impact on subsequent layers.

Datasets

The paper evaluates their method on four diverse datasets and corresponding neural network models:

  • Credit Card Fraud Detection: A binary classification task.
  • KDD Cup '99: A dataset for network intrusion detection.
  • MNIST: A dataset for handwritten digit recognition.
  • CIFAR-10: A dataset for image classification.

These datasets represent a range of complexities, from structured tabular data to images, and include both binary and multi-class classification problems, demonstrating the generalizability of their approach.

Key Insights and Ideas

  • Data-Free Pruning is Crucial: The paper highlights the practical necessity of data-free pruning due to data privacy and regulatory constraints, a gap often overlooked in prior research.
  • Progressive Pruning is Key for Robustness: Aggressive one-shot pruning, while efficient, can be detrimental to robustness when fine-tuning isn't possible. A gradual, iterative approach allows for better control and preservation of critical network functions.
  • Robustness Metrics are Essential for Guiding Pruning: Simply aiming for accuracy preservation isn't enough. Explicitly incorporating robustness metrics (L1-norm and entropy of impact) into the pruning objective is vital for models deployed in adversarial environments.
  • Stochastic Optimization Offers a Robust Solution: Using simulated annealing to navigate the pruning search space helps avoid suboptimal solutions and find a better balance between model compression and robustness.
  • Uniform Impact Distribution is Beneficial: The insight that a uniformly distributed pruning impact across output units is less likely to degrade robustness is a core principle guiding their metric design.

Key Results

  • Significant Robustness Preservation: The method demonstrates substantial robustness preservation, with models retaining at least 50% of their original robustness even after 70% of hidden units are pruned (for models #3 and #4).
  • Favorable Accuracy Retention: All tested models preserved at least 50% of their original accuracy after 50% of hidden units were pruned.
  • Outperforms One-Shot Data-Free Methods: The proposed supervised pruning method significantly outperforms existing one-shot data-free pruning approaches, showing improvements of up to 42% in robustness preservation and 66% in accuracy on specific models.
  • Generalizability: The method shows effectiveness across diverse neural network architectures (MLPs and CNNs) and datasets, indicating its broad applicability.
  • Efficiency: The pruning process can be completed within an acceptable time frame, with multi-class prediction models taking around 8 minutes for 80% pruning in a conservative setting.

Limitations and Future Work

  • Focus on Fully Connected Layers: The current method is primarily designed for fully connected layers. Future work should explore its application to other layer types, such as convolutional, pooling, and normalization layers, and potentially transformer models.
  • Interval Arithmetic Precision: The reliance on interval arithmetic for approximating hidden unit valuations and pruning impact means the precision of these intervals is critical. For ReLU-only models with many layers, magnitude explosion issues might arise.
  • Activation Function Challenges: Pruning models with a mix of convergent (e.g., sigmoid) and non-convergent (e.g., ReLU) activations might be challenging, as convergent activations can reduce the quantitative differences in impact, potentially affecting the sampling criterion's effectiveness.
  • Future Directions: The authors suggest developing additional pruning criteria for diverse layer types and exploring more precise interval approximation or refinement techniques to mitigate magnitude explosion issues. They also propose extending data-free pruning to more layer types, especially over-parameterized ones like 2D convolutional layers.

Summary updated

Abstract

When deploying pre-trained neural network models in real-world applications, model consumers often encounter resource-constraint platforms such as mobile and smart devices. They typically use the pruning technique to reduce the size and complexity of the model, generating a lighter one with less resource consumption. Nonetheless, most existing pruning methods are proposed with the premise that the model after being pruned has a chance to be fine-tuned or even retrained based on the original training data. This may be unrealistic in practice, as the data controllers are often reluctant to provide their model consumers with the original data. In this work, we study the neural network pruning in the data-free context, aiming to yield lightweight models that are not only accurate in prediction but also robust against undesired inputs in open-world deployments. Considering the absence of the fine-tuning and retraining that can fix the mis-pruned units, we replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process. We propose a pruning method based on stochastic optimization that uses robustness-related metrics to guide the pruning process. Our method is implemented as a Python program and evaluated with a series of experiments on diverse neural network models. The experimental results show that it significantly outperforms existing one-shot data-free pruning approaches in terms of robustness preservation and accuracy.
BibTeX
@inproceedings{meng2023paoding,
  author = {Mark Huasong Meng and Guangdong Bai and Sin Gee Teo and Jin Song Dong},
  title = {{Paoding: Supervised Robustness-preserving Data-free Neural Network Pruning}},
  booktitle = {ICECCS},
  year = {2023},
  doi = {10.1109/iceccs59891.2023.00013},
}
Figure