Improving Deep Learning Model Robustness Against Adversarial Attack by Increasing the Network Capacity

Marco Marchetti, Edmond S. L. HoarXiv 2022 · DOI · arXiv

Abstract

Deep learning models are increasingly deployed in security-sensitive environments, yet they remain highly vulnerable to adversarial examples—inputs specifically designed to trigger misclassification. This paper explores the hypothesis that increasing a neural network's capacity, specifically its width, is a fundamental requirement for achieving adversarial robustness. By implementing a Wide ResNet architecture and utilizing aggressive Projected Gradient Descent (PGD) adversarial training, the authors demonstrate a significant improvement in model resilience against strong attacks. However, this robustness comes at the cost of reduced standard accuracy and substantial computational overhead.

Introduction

The rapid integration of Deep Learning (DL) into daily life, often through Machine Learning as a Service (MLaaS) platforms, has expanded the attack surface for adversarial threats. An adversarial example is a piece of data, such as an image, modified by a perturbation so small it is imperceptible to humans but causes a model to misclassify the input with high confidence.

While benchmarking platforms like DeepSec provide tools to evaluate these vulnerabilities, many standard implementations fail to account for the "capacity" required for a model to be truly robust. Following foundational research by Madry et al., this paper investigates whether increasing network capacity and employing more rigorous training parameters can bridge the gap between theoretical security and practical application.

Methodology

The authors extended the DeepSec platform to incorporate advanced architectural and training strategies aimed at maximizing robustness.

Wide ResNet Architecture

The primary architectural modification involved the use of a Wide ResNet. Unlike standard ResNet models, the Wide ResNet increases the number of filters in the internal convolutional layers and the size of the Fully Connected (FC) layer by a factor of 10. This increase in width provides the model with more "expressive power," allowing it to learn the complex decision boundaries necessary to distinguish between clean and adversarial features.

Adversarial Training with PGD

The study utilized Projected Gradient Descent (PGD) as the primary "universal" first-order attack method. PGD iteratively searches for the perturbation that maximizes the model's loss within a predefined budget, denoted by $\epsilon$.

The defense mechanism, PGD Adversarial Training (PAT), involves training the model on these adversarial examples rather than just clean data. A key contribution of this work is the evaluation of a high-perturbation budget ($\epsilon = 8$) for CIFAR-10, contrasting with the more common, lower budget of $\epsilon = 0.1$.

Experimental Results

The models were evaluated on two benchmark datasets: MNIST (handwritten digits) and CIFAR-10 (color images across 10 classes).

Robustness vs. Accuracy Trade-off

The experiments revealed a clear trade-off between a model's performance on clean data and its resilience under attack. As shown in Figure 3, the Wide ResNet achieved a high raw accuracy of 94.16% on the CIFAR-10 dataset. However, when the model was hardened using PAT with a large perturbation budget ($\epsilon = 8$), the accuracy on clean data dropped to 40.25%.

Table showing accuracy of raw vs. defense-enhanced models across different configurations.

This "no free lunch" phenomenon suggests that as a model learns to ignore adversarial noise, it may also lose its ability to distinguish some fine-grained features of the original data.

Attack Effectiveness and Perceptibility

The authors also compared different attack methodologies. While the Carlini-Wagner (CW2) attack is highly effective at achieving a near 100% Misclassification Ratio (MR), the study noted that its perceptibility to human observers (measured via Perturbation Sensitivity Distance) differs from iterative methods like PGD.

Comparison of various attack metrics including Misclassification Ratio (MR) and Perturbation Sensitivity Distance (PSD).

As seen in Figure 1, both PGD and CW2 attacks easily achieve near-total success against undefended models, emphasizing the necessity of robust training techniques in production environments.

Discussion and Limitations

The study confirms that network capacity is a prerequisite for robustness. A standard-sized network often lacks the capacity to learn both the primary classification task and the resistance to adversarial noise simultaneously. By widening the network, the authors provided the necessary "room" for the model to adapt to adversarial training effectively.

However, the practical implementation of these robust models faces significant hurdles:

  1. Computational Cost: Retraining the Wide ResNet on the adversarial-augmented dataset required approximately 5 days (7,000 minutes) on an NVIDIA GeForce 1080 GPU. This makes frequent model updates or large-scale deployment challenging for many organizations.
  2. Accuracy Degradation: The significant drop in standard accuracy when using high $\epsilon$ values remains a major barrier for applications where high precision on clean data is non-negotiable.

Conclusion

Increasing network capacity through wider architectures is a critical step toward building adversarial-resistant deep learning models. While the trade-offs in accuracy and training time are substantial, the use of Wide ResNets combined with aggressive PGD adversarial training provides a path forward for securing DL systems against sophisticated threats. Future research should focus on optimizing these architectures to maintain high natural accuracy while preserving adversarial robustness.

Summary updated

Abstract

Nowadays, we are more and more reliant on Deep Learning (DL) models and thus it is essential to safeguard the security of these systems. This paper explores the security issues in Deep Learning and analyses, through the use of experiments, the way forward to build more resilient models. Experiments are conducted to identify the strengths and weaknesses of a new approach to improve the robustness of DL models against adversarial attacks. The results show improvements and new ideas that can be used as recommendations for researchers and practitioners to create increasingly better DL algorithms.
BibTeX
@misc{marchetti2022improving,
  author = {Marco Marchetti and Edmond S. L. Ho},
  title = {{Improving Deep Learning Model Robustness Against Adversarial Attack by Increasing the Network Capacity}},
  year = {2022},
  eprint = {2204.11357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.11357},
}
Figure