TL;DR
This paper introduces a novel "data-free" neural network pruning method that preserves model robustness, a critical factor for real-world deployments where original training data is unavailable. By treating pruning as a progressive, supervised process guided by robustness metrics and employing stochastic optimization, their method significantly outperforms existing data-free techniques in both accuracy and robustness.
What Problem Does This Paper Address?
Deploying powerful deep learning models on resource-constrained devices like smartphones and IoT devices often requires reducing their size and computational demands. Neural network pruning is a common technique for this, but most existing methods assume the pruned model can be fine-tuned or retrained using the original training data. This assumption is often unrealistic in practice, especially due to data privacy concerns and regulations like GDPR, where data controllers are reluctant to share sensitive training data. This creates a need for "data-free" pruning methods that can reduce model size without access to the original data, while also ensuring the pruned model remains robust against unexpected or adversarial inputs in open-world scenarios.
Methods
The authors propose a supervised, progressive, data-free neural network pruning framework. Instead of aggressive one-shot pruning, they adopt a conservative approach that treats pruning as a gradual process. Key elements include:
- Progressive Pruning: Pruning is done iteratively, cutting off small sets of units and evaluating their impact at each step to minimize mis-pruning of crucial units.
- Robustness-Guided Pruning: The pruning process is guided by metrics that quantify the model's robustness. Specifically, they use L1-norm (measuring the overall scale of pruning impact on the model's output) and entropy (measuring the similarity of pruning impact across output units) to ensure the impact is uniformly distributed.
- Stochastic Optimization with Simulated Annealing: To identify which units to prune, they employ a two-stage approach:
- Candidate Prioritizing: Units are prioritized based on their "saliency," a measure of their importance.
- Stochastic Sampling: A simulated annealing algorithm is used to sample candidates, guided by the robustness metrics. This helps avoid getting stuck in local optima and finds units with minimal impact on robustness.
- Primitive Pruning Operation: When pruning a unit (nominee), they don't simply delete it. Instead, they zero out its connections and adjust the parameters of another unit (delegate) in the same layer to cover its duty, minimizing the impact on subsequent layers.
Datasets
The paper evaluates their method on four diverse datasets and corresponding neural network models:
- Credit Card Fraud Detection: A binary classification task.
- KDD Cup '99: A dataset for network intrusion detection.
- MNIST: A dataset for handwritten digit recognition.
- CIFAR-10: A dataset for image classification.
These datasets represent a range of complexities, from structured tabular data to images, and include both binary and multi-class classification problems, demonstrating the generalizability of their approach.
Key Insights and Ideas
- Data-Free Pruning is Crucial: The paper highlights the practical necessity of data-free pruning due to data privacy and regulatory constraints, a gap often overlooked in prior research.
- Progressive Pruning is Key for Robustness: Aggressive one-shot pruning, while efficient, can be detrimental to robustness when fine-tuning isn't possible. A gradual, iterative approach allows for better control and preservation of critical network functions.
- Robustness Metrics are Essential for Guiding Pruning: Simply aiming for accuracy preservation isn't enough. Explicitly incorporating robustness metrics (L1-norm and entropy of impact) into the pruning objective is vital for models deployed in adversarial environments.
- Stochastic Optimization Offers a Robust Solution: Using simulated annealing to navigate the pruning search space helps avoid suboptimal solutions and find a better balance between model compression and robustness.
- Uniform Impact Distribution is Beneficial: The insight that a uniformly distributed pruning impact across output units is less likely to degrade robustness is a core principle guiding their metric design.
Key Results
- Significant Robustness Preservation: The method demonstrates substantial robustness preservation, with models retaining at least 50% of their original robustness even after 70% of hidden units are pruned (for models #3 and #4).
- Favorable Accuracy Retention: All tested models preserved at least 50% of their original accuracy after 50% of hidden units were pruned.
- Outperforms One-Shot Data-Free Methods: The proposed supervised pruning method significantly outperforms existing one-shot data-free pruning approaches, showing improvements of up to 42% in robustness preservation and 66% in accuracy on specific models.
- Generalizability: The method shows effectiveness across diverse neural network architectures (MLPs and CNNs) and datasets, indicating its broad applicability.
- Efficiency: The pruning process can be completed within an acceptable time frame, with multi-class prediction models taking around 8 minutes for 80% pruning in a conservative setting.
Limitations and Future Work
- Focus on Fully Connected Layers: The current method is primarily designed for fully connected layers. Future work should explore its application to other layer types, such as convolutional, pooling, and normalization layers, and potentially transformer models.
- Interval Arithmetic Precision: The reliance on interval arithmetic for approximating hidden unit valuations and pruning impact means the precision of these intervals is critical. For ReLU-only models with many layers, magnitude explosion issues might arise.
- Activation Function Challenges: Pruning models with a mix of convergent (e.g., sigmoid) and non-convergent (e.g., ReLU) activations might be challenging, as convergent activations can reduce the quantitative differences in impact, potentially affecting the sampling criterion's effectiveness.
- Future Directions: The authors suggest developing additional pruning criteria for diverse layer types and exploring more precise interval approximation or refinement techniques to mitigate magnitude explosion issues. They also propose extending data-free pruning to more layer types, especially over-parameterized ones like 2D convolutional layers.
Summary updated