Abstract
Adversarial training (AT) is the gold standard for securing deep learning models, but its computational cost is notoriously high. This paper introduces GAAT (Adversarial Training with Gradient Approximation), a method that uses Taylor series expansions to approximate adversarial losses, slashing training time by up to 60% while maintaining comparable robustness.
Introduction
Deep neural networks are famously fragile; adding a tiny, imperceptible amount of adversarial noise to an image can trick a classifier into misidentification. To mitigate this, researchers use Adversarial Training (AT), where the model is trained on these perturbed examples. However, the most effective method, Projected Gradient Descent (PGD) based training, requires an "inner loop" of multiple iterations for every training step, making it 10 to 15 times slower than standard training. GAAT seeks to bridge this efficiency gap without sacrificing defensive strength.
The GAAT Method: Gradient Approximation
The core innovation of GAAT is leveraging the Taylor Series to approximate the adversarial loss. Since adversarial perturbations ($\delta$) are designed to be small, the adversarial loss can be approximated using a second-order expansion: $$\ell(x + \delta) \approx \ell(x) + \delta^T J(x) + \frac{1}{2} \delta^T H(x) \delta$$ where $J$ is the Jacobian and $H$ is the Hessian.
By differentiating this, we get a simplified gradient: $\nabla_\delta \ell(x + \delta) \approx J(x) + H(x)\delta$. To maintain efficiency, the Hessian is approximated using Gauss-Newton or diagonal methods. In standard PGD, $T$ full backpropagation steps are performed. In GAAT, the Jacobian and Hessian are computed once per batch, followed by $T$ fast Taylor-based iterations.
Experimental Evaluation
GAAT was evaluated on MNIST, CIFAR-10, and CIFAR-100 using architectures like VGGNet-16, ResNet-18, ResNet-50, and WideResNet-28x4. The study focused on the trade-off between training speed and the resulting model's robustness against various attack strengths.
Key Results and Discussion
GAAT achieves massive speedups. For a WideResNet-28x4 model on CIFAR-10, training time was reduced from 1,345 minutes to 537 minutes—a 60% reduction. Robustness is largely preserved; for ResNet-50, adversarial accuracy only dropped from 46.26% to 44.54%, while natural accuracy slightly improved.
The method also shows strong synergy with Delayed Adversarial Training (DAT). By starting with natural training and switching to GAAT later, total training time can be reduced by up to 80% compared to standard PGD training.
Conclusion
GAAT offers a practical path toward efficient adversarial robustness. While a small "approximation gap" exists due to omitting higher-order Taylor terms, the computational savings are substantial. Future work may explore more sophisticated Hessian approximations to further close the robustness gap without increasing the training budget.
Summary updated