Hardening Deep Neural Networks via Adversarial Model Cascades

Deepak Vijaykeerthy, Anshuman Suri, Sameep Mehta, Ponnurangam KumaraguruIJCNN 2018 · DOI

Abstract

Deep Neural Networks (DNNs) are susceptible to adversarial perturbations that compromise their reliability in security-critical applications. We propose Adversarial Model Cascades (AMC), a sequential training framework designed to harden DNNs against a diverse set of adversarial attacks. By employing parameter transfer and a cumulative weighted loss function, AMC effectively generalizes robustness across multiple attack vectors. Our evaluation on MNIST, SVHN, and CIFAR-10 demonstrates that AMC achieves superior empirical robustness compared to standard adversarial training without degrading performance on clean data.

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in various computer vision tasks, yet they remain notoriously fragile. Small, human-imperceptible perturbations to an input—known as adversarial examples—can easily trick a model into making confident, incorrect predictions. This vulnerability poses a significant risk for real-world deployments.

Adversarial examples across different datasets. The perturbations are often human-imperceptible yet lead to misclassification.

While "Adversarial Training"—the process of injecting malicious examples into the training set—is a popular defense, it suffers from several limitations. Models often overfit to specific attacks, failing to resist different or stronger methods. Furthermore, many defenses do not scale well from simple datasets like MNIST to more complex ones like CIFAR-10, and they remain vulnerable to adaptive adversaries who can bypass defenses in black-box settings using proxy models.

Methodology: Adversarial Model Cascades (AMC)

The core innovation of this work is the Adversarial Model Cascade (AMC). Rather than training a single model on a static pool of adversarial examples, we propose training a sequence of models ($M^0, M^1, \dots, M^E$) to achieve incremental hardening.

Sequential Hardening and Parameter Transfer

Each model $M^i$ in the cascade is initialized with the weights of the preceding model $M^{i-1}$. This parameter transfer mechanism serves two purposes: it significantly accelerates the training process and provides a stable starting point that preserves the robustness features learned in earlier stages.

Multi-Task Loss Function

To prevent the "forgetting" problem—where a model becomes vulnerable to an old attack while learning to resist a new one—we utilize a weighted loss function. The total loss is a sum of the loss on clean data and a weighted combination of adversarial losses from the current attack and all previous attacks in the cascade. This approach treats adversarial robustness as a multi-task learning problem.

Integration with Feature Squeezing

We also explore the synergy between AMC and Feature Squeezing (e.g., bit-depth reduction). While Feature Squeezing is often insufficient as a standalone defense, it acts as a powerful "force multiplier" when combined with AMC. It effectively denoises high-intensity attacks that the model may not have fully mastered during the cascade training.

Experimental Evaluation

We evaluated AMC on three standard benchmarks: MNIST, SVHN, and CIFAR-10. To simulate realistic black-box scenarios, we trained "local proxy" models on independent datasets (e.g., STL-10 for CIFAR-10 attacks) to craft adversarial examples for the target model.

Key Results

AMC consistently outperformed standard adversarial training across all tested datasets. In white-box evaluations, AMC significantly reduced error rates compared to undefended models and existing defense baselines.

White-box attack error rates (%) for various models. AMC significantly reduces error rates compared to the undefended baseline across all datasets.
  • Robustness Gains: AMC increased empirical accuracy under attack by an average of 6.2% for MNIST, 5.1% for SVHN, and 2.6% for CIFAR-10 compared to the best existing adversarial training methods.
  • Performance on Clean Data: Crucially, AMC maintained clean-data classification performance comparable to state-of-the-art undefended models, avoiding the accuracy trade-off common in other defenses.
  • Defense Synergy: On CIFAR-10, AMC reduced the error rate against the PGM attack from 97.2% to 62.5%. When combined with Feature Squeezing, this error rate dropped further to 38.2%.

Discussion and Future Work

Our results indicate that the order of attacks in the cascade is a critical factor; introducing stronger attacks earlier can improve final robustness but requires careful scheduling to avoid overfitting. While AMC is computationally more intensive than single-stage training, the use of parameter transfer mitigates this cost.

Future research will focus on scaling AMC to high-resolution datasets like ImageNet and exploring its effectiveness against an even broader diversity of evolving adversarial techniques. By treating robustness as a sequential learning process, AMC provides a robust framework for building more resilient deep learning systems.

Summary updated

BibTeX
@inproceedings{vijaykeerthy2018hardening,
  author = {Deepak Vijaykeerthy and Anshuman Suri and Sameep Mehta and Ponnurangam Kumaraguru},
  title = {{Hardening Deep Neural Networks via Adversarial Model Cascades}},
  booktitle = {IJCNN},
  year = {2018},
  doi = {10.1109/ijcnn.2019.8851970},
}
Figure