Role of Optimizer on Network Fine-tuning for Adversarial Robustness (Student Abstract)

Akshay Agarwal, Mayank Vatsa, Richa Singh

Abstract

Deep Neural Networks (DNNs) are susceptible to adversarial attacks, and traditional adversarial training is computationally prohibitive for many researchers. This work proposes a cost-effective defense using short-term network fine-tuning with localized, random data augmentation. By perturbing local image patches and selecting appropriate optimizers—specifically RMSProp—the proposed method achieves significant robustness against gradient-based attacks (FGSM, IFGSM, PGD) while maintaining or improving clean accuracy. The approach offers a 60x speedup over standard adversarial training while often outperforming it in robustness metrics.

Introduction

Deep Neural Networks (DNNs) have revolutionized computer vision but remain notoriously vulnerable to adversarial attacks—small, intentional perturbations that cause models to misclassify images. While "Adversarial Training" (AT), which involves training a model on adversarial examples from scratch, is a common defense, it is incredibly resource-intensive. This "resource barrier" often prevents academic institutions with limited compute power from developing robust models.

This paper addresses this challenge by investigating whether robust models can be achieved through a lightweight fine-tuning approach rather than training from scratch. The authors focus on targeting local regions of an image where neurons exhibit the highest sensitivity to perturbations.

Methodology

The proposed defense mechanism utilizes a randomized sampling-based data augmentation strategy applied during a brief fine-tuning phase.

Overview of the proposed defense mechanism involving local patch modification and stochastic data augmentation.

Local Patch Modification and Stochastic Selection

Instead of altering the entire image, the method selects $w \times w$ patches (typically centered). This is based on the observation that individual neurons are often sensitive to specific local features. The researchers curate a pool of 10 operations ($K$), including rotation, translation, flipping, noise addition, and pixel masking (occlusion). For each image, a random number $\lambda$ is drawn from a uniform distribution $U(10)$ to select a transformation $q$ from the pool. This "intelligent randomness" helps break the model's over-reliance on specific training distributions.

Fine-tuning and Optimizer Analysis

The network is fine-tuned for a short period (approximately 20 minutes) using a combination of original clean images and these locally modified images. A critical component of this study is the comparison of different optimizers—specifically Adam and RMSProp—to determine their impact on the final robustness of the model.

Results and Analysis

The approach was evaluated on three benchmarks: Fashion-MNIST (F-MNIST), CIFAR-10, and Imagenette. The defense was tested against white-box attacks including the Fast Gradient Sign Method (FGSM), Iterative FGSM (IFGSM), and Projected Gradient Descent (PGD).

Comparison of model accuracy under various attacks across three datasets, highlighting the performance of Adam vs. RMSProp optimizers.

Optimizer Performance

A key finding is that the choice of optimizer during fine-tuning is critical. While Adam is a common default, RMSProp provided superior robustness on complex datasets like CIFAR-10 and Imagenette. For instance, on CIFAR-10, RMSProp reached 60.37% accuracy against PGD attacks, while Adam only reached 34.45%.

Efficiency and Robustness Gains

The proposed fine-tuning took only 20 minutes, compared to the 20 hours required for full adversarial training on a standard GPU (1080ti). Despite this 60x speedup, the results often surpassed full adversarial training. On CIFAR-10, the proposed method achieved 60.37% accuracy against PGD, significantly higher than the 37.93% achieved by standard PGD-based Adversarial Training. Furthermore, unlike many defenses that degrade clean accuracy, this method improved clean image performance by 3% on CIFAR-10.

Conclusion

This research demonstrates that localized, stochastic data augmentation combined with the right optimizer (RMSProp) can provide a highly efficient and effective defense against adversarial attacks. By focusing on local vulnerabilities and leveraging lightweight fine-tuning, robust DNNs can be developed without the massive computational overhead of traditional adversarial training. Future work may involve automating the selection of patch locations and transformations to further enhance the "intelligence" of the defense mechanism.

Summary updated

Abstract

The solutions proposed in the literature for adversarial robustness are either not effective against the challenging gradient-based attacks or are computationally demanding, such as adversarial training. Adversarial training or network training based data augmentation shows the potential to increase the adversarial robustness. While the training seems compelling, it is not feasible for resource-constrained institutions, especially academia, to train the network from scratch multiple times. The two fold contributions are: (i) providing an effective solution against white-box adversarial attacks via network fine-tuning steps and (ii) observing the role of different optimizers towards robustness. Extensive experiments are performed on a range of databases, including Fashion-MNIST and a subset of ImageNet. It is found that the few steps of network fine-tuning effectively increases the robustness of both shallow and deep architectures. To know other interesting observations, especially regarding the role of the optimizer, refer to the paper.
BibTeX
@inproceedings{agarwal2021role,
  author = {Akshay Agarwal and Mayank Vatsa and Richa Singh},
  title = {{Role of Optimizer on Network Fine-tuning for Adversarial Robustness (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17869},
}
Figure