VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch

Sawinder Kaur, Yi Xiao, Asif SalekinAAAI 2024 · DOI

TL;DR

This paper introduces VeriCompress, a novel tool that automates the creation of verified robust and compressed neural networks from scratch. It significantly reduces training time and computational resources compared to existing methods, making safety-critical AI deployable on resource-constrained edge devices.

What Problem Does This Paper Address?

Neural networks (NNs) are increasingly deployed on edge devices for safety-critical applications like autonomous driving and medical diagnosis. However, NNs are notoriously fragile, susceptible to imperceptible input perturbations that can lead to catastrophic inference errors. While methods exist to make NNs robust (e.g., adversarial training and verified robustness), these often come at the cost of reduced accuracy and significantly increased model size and computational demands. This creates a dilemma: deploy accurate but vulnerable NNs, or robust but resource-intensive NNs. Furthermore, existing robust training methods often require pre-training a large, dense model, which is computationally expensive and time-consuming. The challenge is to develop NNs that are both accurate and verifiably robust, while also being small and efficient enough for deployment on resource-limited platforms, all without the prohibitive cost of traditional methods.

Methods

VeriCompress tackles this challenge by automating the search and training of compressed, verified robust neural networks from scratch. The core idea is to start with a randomly initialized, potentially over-parameterized "backbone" architecture and a target parameter budget. VeriCompress then employs a dynamic sparse training paradigm.

Key aspects of the method include:

  • Parameter-Budget-Aware Architecture Search: Instead of pruning an already trained dense model, VeriCompress searches for a sub-network within the backbone architecture that meets the specified parameter budget.
  • Training from Scratch: This eliminates the need for a pre-trained dense model, significantly reducing training time and resource overhead.
  • Dynamic Sparse Training: The approach allows elements (kernels/nodes) of the network to be activated or dormant. Importantly, even dormant elements have their gradients computed during the backward pass, enabling exploration of new network structures.
  • ε-Scheduler: A perturbation scheduler gradually increases the adversarial perturbation budget (ε) during training, which is crucial for preventing issues like bound explosion and ensuring effective learning of robustness.
  • CROWN-IBP as Implicit Regularizer: The training loss function, based on verified robustness principles (specifically CROWN-IBP), implicitly penalizes large parameter magnitudes. This naturally leads to many parameters becoming close to zero, making them candidates for deactivation without significant loss of generalizability.
  • Layer-wise Parameter Deactivation: To determine which parts of the network to prune, VeriCompress uses the L2 norm of kernels/nodes. This is done layer-wise to handle varying parameter counts per kernel/node. The importance of a kernel/node is determined by the norm of its associated parameters.

Datasets

The paper evaluates VeriCompress on several standard datasets:

  • MNIST: A handwritten digit dataset.
  • CIFAR-10: A dataset of 10 common object classes.
  • SVHN (Street View House Numbers): A dataset of house numbers.
  • Pedestrian Detection Dataset: A dataset specifically for identifying people and people-like objects.

Key Insights and Ideas

  • "Scratch" is the Way to Go: Training verified robust compressed networks from scratch, rather than pruning pre-trained models, is significantly more efficient and effective. This bypasses a major bottleneck in existing methods.
  • Implicit Regularization is Powerful: The verified robustness training objective itself naturally encourages sparsity by pushing parameter magnitudes towards zero. This means explicit sparsity-inducing regularization terms are often unnecessary.
  • Layer-wise Importance Matters: Deactivating network elements (kernels/nodes) based on their importance (L2 norm) in a layer-wise manner is a practical approach for structured pruning.
  • Automation is Key for Usability: VeriCompress automates the complex process of finding suitable compressed architectures, making verified robustness accessible to a wider range of users, not just AI experts.

Key Results

  • Improved Accuracy and Robustness: VeriCompress achieves an average increment of 15.1% in Standard Accuracy and 9.8% in Verified Accuracy compared to state-of-the-art baselines (Hydra and FaShapley).
  • Faster Training: VeriCompress trains models 2-3 times faster than existing structured pruning approaches.
  • Significant Resource Reduction: Compressed models require 5-8 times less memory and 2-4 times less inference time when deployed on a resource-constrained platform like a Google Pixel 6, compared to their dense counterparts.
  • Scalability and Generalizability: The tool demonstrates effectiveness across various network architectures (4-layer CNN, 7-layer CNN, ResNet, DenseNet, ResNext) and datasets, indicating broad applicability.

Limitations and Future Work

  • Resource Reduction vs. Generalizability Trade-off: The paper notes that while resource reductions are significant, they come at the cost of some reduced generalizability, which is discussed in Appendix D.
  • Reproducibility of Baselines: The authors mention they could not reproduce the results for the FaShapley baseline with the provided code, necessitating comparison with published results.
  • Future Work: The paper suggests that VeriCompress can be extended to other types of neural network architectures and applications. The authors also highlight the potential for further optimization of hyperparameters and exploration of different pruning strategies.

Summary updated

BibTeX
@inproceedings{kaur2024vericompress,
  author = {Sawinder Kaur and Yi Xiao and Asif Salekin},
  title = {{VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30327},
}
Figure