AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient

Farzad Nikfam, Alberto Marchisio, Maurizio Martina, Muhammad Shafique

Abstract

Adversarial training (AT) is a primary defense mechanism for securing Deep Neural Networks (DNNs) against malicious perturbations. However, the computational overhead of AT is significant, often requiring substantially longer training times than standard methods. This paper introduces AccelAT, a framework designed to accelerate adversarial training by dynamically adjusting the learning rate based on the "accuracy gradient." By monitoring performance plateaus and automatically tuning hyperparameters, AccelAT achieves up to a 2x speedup compared to existing techniques while maintaining or improving model robustness. Our evaluation across CIFAR10 and CIFAR100 datasets demonstrates that AccelAT offers a more efficient path toward developing secure DNN-based applications.

Introduction

Deep Neural Networks are vulnerable to adversarial attacks—small, often imperceptible perturbations to input data that cause models to misclassify samples with high confidence. For instance, an attacker might add noise to an image of a strawberry, causing a model to identify it as a chestnut. Adversarial Training (AT), which incorporates these malicious samples into the training process, remains the most effective defense.

Despite its efficacy, AT is notoriously slow, typically taking 3x to 30x longer than standard training. This bottleneck hinders the rapid development and deployment of robust models. The AccelAT framework addresses this challenge by optimizing the learning rate (LR) schedule—the most critical hyperparameter in training—using a performance-driven approach rather than a static temporal schedule.

The AccelAT Framework

The core innovation of AccelAT is its treatment of the training process as a dynamic system. Rather than following a rigid, pre-defined LR schedule (such as a fixed step-down or cyclical policy), AccelAT monitors the model's progress in real-time.

1. Learning Rate Finder

Before the main training phase, the framework employs an "LR Finder" to identify the maximum stable learning rate. This ensures that the training begins at an optimal velocity without risking divergence.

2. Adaptive Scheduling via Accuracy Gradient

AccelAT utilizes an "accuracy gradient" logic to manage the LR during training. The framework monitors validation accuracy over a set number of epochs ($n$). If the improvement in accuracy falls below a specific threshold ($\Delta_{acc}$), the system identifies a performance plateau. At this point, the framework automatically reduces the learning rate by a reduction factor ($p$).

This adaptive intervention allows the model to maintain a high learning rate for as long as it is making significant progress, only "zooming in" with a lower LR to fine-tune robustness when necessary.

Experimental Methodology

The authors evaluated AccelAT using two benchmark datasets: CIFAR10 and CIFAR100. The framework was tested across multiple architectures, including ResNet-20, ResNet-50, and MobileNet. To assess robustness, the models were subjected to three common adversarial attacks:

  • LinfPGD (Projected Gradient Descent)
  • FGSM (Fast Gradient Sign Method)
  • DeepFool

Key Findings

1. Significant Acceleration

Experimental results show that AccelAT can reach comparable or superior accuracy in approximately half the training epochs required by state-of-the-art methods like Free Adversarial Training (FAT). This 2x speedup directly translates to reduced GPU energy consumption and faster iteration cycles for researchers.

2. Enhanced Robustness

Beyond speed, AccelAT often yielded more robust models. On the CIFAR100 dataset using MobileNet, AccelAT achieved up to 8% higher adversarial accuracy than standard fixed-schedule methods. By dynamically responding to the model's learning state, the framework appears to find better local minima that enhance generalization against attacks.

3. Scalability to Complexity

The framework demonstrated its most significant advantages on complex tasks (CIFAR100) and efficient architectures (MobileNet). This suggests that as the complexity of the data or the constraints of the model increase, adaptive accuracy-based scheduling becomes increasingly superior to static schedules.

Limitations and Future Directions

While AccelAT automates the learning rate adjustment, it introduces its own hyperparameters: the accuracy threshold ($\Delta_{acc}$), the monitoring window ($n$), and the reduction factor ($p$). Currently, these parameters require some initial tuning based on the specific dataset.

Future research will focus on:

  • Multi-parameter Automation: Expanding the framework to simultaneously adjust batch size and momentum.
  • Large-scale Benchmarking: Testing AccelAT on massive datasets like ImageNet and extending the methodology to domains beyond computer vision, such as natural language processing.

Conclusion

AccelAT represents a significant step forward in making adversarial training practical for real-world applications. By leveraging the accuracy gradient to automate learning rate scheduling, the framework halves training time while simultaneously boosting model security, addressing two of the most critical challenges in modern DNN development.

Reference: Nikfam, F., et al. "AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient." IEEE Access / arXiv:2210.06888v1, 2022.

Summary updated

Abstract

Adversarial training is exploited to develop a robust Deep Neural Network (DNN) model against the malicious altered data. These attacks may have catastrophic effects on DNN models but are indistinguishable for a human being. For example, an external attack can modify an image adding noises invisible for a human eye, but a DNN model misclassified the image. A key objective for developing robust DNN models is to use a learning algorithm that is fast but can also give model that is robust against different types of adversarial attacks. Especially for adversarial training, enormously long training times are needed for obtaining high accuracy under many different types of adversarial samples generated using different adversarial attack techniques. This paper aims at accelerating the adversarial training to enable fast development of robust DNN models against adversarial attacks. The general method for improving the training performance is the hyperparameters fine-tuning, where the learning rate is one of the most crucial hyperparameters. By modifying its shape (the value over time) and value during the training, we can obtain a model robust to adversarial attacks faster than standard training. First, we conduct experiments on two different datasets (CIFAR10, CIFAR100), exploring various techniques. Then, this analysis is leveraged to develop a novel fast training methodology, AccelAT, which automatically adjusts the learning rate for different epochs based on the accuracy gradient. The experiments show comparable results with the related works, and in several experiments, the adversarial training of DNNs using our AccelAT framework is conducted up to 2 times faster than the existing techniques. Thus, our findings boost the speed of adversarial training in an era in which security and performance are fundamental optimization objectives in DNN-based applications.
BibTeX
@inproceedings{nikfam2022accelat,
  author = {Farzad Nikfam and Alberto Marchisio and Maurizio Martina and Muhammad Shafique},
  title = {{AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=2g7CgiCgxcA},
}
Figure