Pruning Adversarially Robust Neural Networks without Adversarial Examples

Tong Jian, Zifeng Wang, Yanzhi Wang, Jennifer G. Dy, Stratis Ioannidis

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success across various domains, yet they remain notoriously vulnerable to adversarial attacks—subtle, often imperceptible perturbations to input data that can lead to catastrophic misclassifications. The standard defense, adversarial training, involves generating malicious examples during the training loop. While effective, this process is computationally expensive, often increasing training time by 3x to 30x.

Furthermore, deploying these robust models on resource-constrained edge devices requires model compression, typically through pruning. However, a significant dilemma arises: naive pruning using natural images destroys a model's robustness, while "robust pruning"—which incorporates adversarial examples—is prohibitively slow and must be constantly updated against evolving attack methods. This paper introduces PwoA (Pruning without Adversarial examples), a framework designed to compress robust networks while maintaining their defensive capabilities using only clean, natural images.

Methodology

The PwoA framework avoids the generation of adversarial examples during the pruning process by leveraging three core components to preserve the decision boundaries of a pre-trained robust model.

1. Robustness-Preserving Self-Distillation

Instead of training the pruned "student" model against hard labels (e.g., one-hot encoded categories), PwoA employs self-distillation. The student model is trained to mimic the "soft labels" or probability outputs of the original, unpruned "teacher" model. By matching the teacher's output distribution, the student inherits the robust decision boundaries established during the teacher's initial adversarial training.

2. HSIC Bottleneck Regularization

To further harden the model without exposing it to adversarial noise, the authors utilize the Hilbert-Schmidt Independence Criterion (HSIC). This information-theoretic tool serves two purposes:

  • Input-Representation Independence: It minimizes the dependency between the input data and internal representations, effectively filtering out non-essential features and potential adversarial perturbations.
  • Representation-Label Dependence: It maximizes the dependency between internal representations and the correct labels, ensuring that the model retains features critical for classification.

3. ADMM Optimization

Pruning is fundamentally a combinatorial optimization problem—determining the optimal subset of weights to remove. PwoA utilizes the Alternating Direction Method of Multipliers (ADMM) to decompose this complex problem into smaller, manageable sub-problems. This ensures the model achieves high sparsity targets (e.g., 90% or higher) while adhering to the constraints of the distillation and HSIC objectives.

Experimental Evaluation

The framework was evaluated on three standard computer vision benchmarks: MNIST, CIFAR-10, and CIFAR-100. The pruned models were subjected to rigorous testing against five state-of-the-art attacks, including Projected Gradient Descent (PGD) and AutoAttack (AA), the latter being the current gold standard for evaluating adversarial robustness.

Key Findings

  • Efficiency Gains: PwoA achieves a 4x to 7x training speedup compared to existing state-of-the-art (SOTA) adversarial pruning methods like HYDRA or AdvPrune. This is primarily due to the elimination of the adversarial example generation step during pruning.
  • Robustness at Scale: The framework demonstrates remarkable resilience to high sparsity. On CIFAR-10, PwoA-pruned models at a 16x pruning rate (93.75% sparsity) outperformed smaller models trained from scratch with adversarial examples by over 10% in robustness.
  • The Role of Classification Loss: Interestingly, the authors discovered that including standard cross-entropy loss (training on hard labels) during the pruning phase actually decreased robustness. PwoA achieves superior results by relying exclusively on distillation and HSIC regularization.
  • Attack-Agnostic Defense: Because PwoA does not train against a specific attack vector, it generalizes well across different types of adversarial threats, maintaining higher robustness against AutoAttack on CIFAR-100 compared to SOTA methods when training time is restricted.

Discussion and Limitations

While PwoA offers a significant leap in efficiency, it is inherently dependent on the quality of the "teacher" model. The pruned student cannot exceed the robustness of the initial pre-trained model. Additionally, this research focuses on unstructured pruning (removing individual weights). While this reduces the theoretical parameter count, it does not always translate to immediate hardware acceleration without specialized libraries.

Future work may explore structured pruning (removing entire filters or neurons) to achieve direct hardware speedups, as well as the integration of weight quantization to further compress robust models for extreme edge deployment.

Conclusion

PwoA demonstrates that adversarial examples are not a prerequisite for pruning robust neural networks. By combining self-distillation with HSIC-based regularization, researchers can produce highly efficient, sparse models that retain their defensive integrity at a fraction of the traditional computational cost.

Summary updated

BibTeX
@inproceedings{jian2022pruning,
  author = {Tong Jian and Zifeng Wang and Yanzhi Wang and Jennifer G. Dy and Stratis Ioannidis},
  title = {{Pruning Adversarially Robust Neural Networks without Adversarial Examples}},
  booktitle = {ICDM},
  year = {2022},
  doi = {10.1109/icdm54844.2022.00120},
}
Figure