Abstract
Deep neural networks (DNNs) are often too large for deployment on resource-constrained mobile and IoT devices, necessitating model pruning. However, traditional pruning methods typically require access to the original training data for fine-tuning—a luxury often unavailable due to privacy regulations like GDPR or proprietary restrictions. Furthermore, aggressive pruning frequently compromises a model's robustness against adversarial attacks. This paper introduces Paoding, a supervised, data-free pruning framework that treats model reduction as a progressive evolution guided by simulated annealing. By utilizing a "delegate" neuron strategy and robustness-guided metrics, Paoding preserves both accuracy and adversarial stability, significantly outperforming traditional one-shot pruning techniques.
Introduction
The proliferation of deep learning in edge computing has created a critical need for efficient model compression. While pruning effectively removes redundant parameters, it introduces two primary hurdles: the Data Gap and the Robustness Gap. Most existing methods rely on the original training data to recover performance lost during pruning. In data-free scenarios, engineers often resort to "one-shot" pruning, which can cause irreversible damage to the model's internal logic and make it highly vulnerable to small, malicious perturbations.
Paoding addresses these challenges by reframing pruning not as a single surgical strike, but as a supervised, iterative process. This approach ensures that the model's decision boundaries remain stable even as its architecture shrinks.
The Paoding Framework
The core innovation of Paoding lies in its three-pronged approach to maintaining model integrity without external data:
1. Saliency-based Primitive Pruning
Rather than simply deleting a neuron (the "nominee"), Paoding identifies a "delegate" neuron within the same layer that performs a functionally similar role. The framework then adjusts the delegate's weights to compensate for the removed neuron's contribution. This "duty coverage" strategy minimizes the immediate shock to the network's signal flow, preventing the catastrophic forgetting typically seen in data-free pruning.
2. Stochastic Optimization via Simulated Annealing
To navigate the complex search space of optimal neuron configurations, Paoding employs Simulated Annealing. This stochastic optimization algorithm allows the system to occasionally accept sub-optimal pruning steps in the early stages. By doing so, it avoids getting trapped in local mathematical minima, eventually converging on a global configuration that balances size and performance more effectively than greedy algorithms.
3. Robustness-Guided Metrics
The pruning process is governed by a specialized "Energy" function designed to preserve the model's output distribution. This function focuses on two metrics:
- Scale (L1-norm): This measures the total magnitude of change in the output. By minimizing this change, the framework ensures the pruned model's outputs remain close to the original.
- Entropy: This measures how the pruning impact is distributed across output classes. Maintaining uniform impact prevents the model from becoming biased or flipping its final classification decision under slight pressure.
Experimental Evaluation
The researchers evaluated Paoding across four diverse datasets to test its versatility in real-world scenarios:
- Credit Card Fraud Detection: A 4-layer MLP for binary classification.
- KDD Cup ’99: A 6-layer MLP for network intrusion detection.
- MNIST: A 5-layer MLP for digit recognition.
- CIFAR-10: A 13-layer CNN for complex image classification.
Key Results
Paoding demonstrated a significant performance leap over baseline saliency-based one-shot methods:
- Fidelity Preservation: All models maintained at least 50% of their original accuracy even after 50% of their hidden units were removed.
- Robustness Resilience: Models retained 50% of their original adversarial robustness even at a 70% pruning rate.
- Comparative Superiority: Compared to existing data-free approaches, Paoding improved robustness preservation by up to 42% and accuracy by up to 66%.
- Operational Efficiency: Despite its iterative nature, the process is highly efficient. Pruning complex multi-class models took less than 8 minutes on standard consumer hardware (NVIDIA RTX 3060Ti).
Discussion and Future Work
Paoding proves that robustness can be "baked into" the pruning process. By specifically monitoring how the output distribution changes, the method can remove "jittery" or sensitive neurons, sometimes even slightly improving robustness during the compression process.
However, some limitations remain. The current framework is primarily optimized for fully connected (FC) layers and ReLU activation functions. Future research will focus on extending this logic to 2D convolutional layers and Transformer architectures. Additionally, the researchers aim to address "magnitude explosion" in very deep networks, where the interval arithmetic used to estimate neuron values can become less precise.
Conclusion
Paoding represents a significant step forward for data-free model compression. By combining functional delegation with stochastic optimization, it bridges the gap between efficiency and security, allowing developers to deploy compact, robust neural networks without compromising data privacy.
Summary updated