TL;DR
This paper introduces "AttackRider," a novel method to accelerate adversarial training (AT) by exploiting under-utilized GPU resources. Instead of processing minibatches one by one for attack generation, AttackRider processes multiple minibatches in a single, overhead-dominated attack call, significantly reducing training time without sacrificing robustness.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are susceptible to adversarial attacks, where small, imperceptible perturbations to input data can cause misclassifications. Adversarial training (AT) is a common defense mechanism that involves generating these adversarial examples during training to make the model more robust. However, AT is notoriously slow because it requires generating worst-case adversarial examples for each minibatch, often involving multiple gradient descent steps. Existing methods to speed up AT either reduce the number of attack steps or the amount of data processed, but these can compromise model robustness or fail to address the root cause of the slowdown: GPU under-utilization. This paper tackles the problem of AT's computational cost by identifying and exploiting scenarios where GPUs are not fully utilized during the attack generation phase.
Methods
The core innovation of AttackRider lies in its concept of "overhead throughput" ($O^*$). The authors observe that for certain model and GPU configurations, the runtime of the attack generation function ($Atk(X, K)$) is dominated by a fixed overhead (e.g., kernel launch, memory allocation) until the input data size ($|X|$) reaches a certain threshold, $O^*$. When the data size is below $O^*$, the GPU is under-utilized. AttackRider leverages this by processing multiple minibatches ($b$) in a single $Atk(X, K)$ call, as long as the total data size ($b \times m$, where $m$ is the base AT's minibatch size) remains below $O^*$. This allows $b$ minibatches to "hitch a ride" on the overhead time of a single attack generation.
To maintain model performance, AttackRider employs two key techniques:
- Shuffling: The combined batch of adversarial and natural examples is shuffled.
- Dividing: The shuffled batch is then divided back into the original minibatch size for the model update step.
This ensures that the model update frequency remains consistent with the original base AT, preventing the degradation in generalization often seen with larger effective batch sizes. AttackRider is designed as a general framework that can be applied to existing AT algorithms.
Datasets
The paper evaluates AttackRider on several standard datasets:
- Image Datasets: CIFAR-10, CIFAR-100, and TinyImageNet.
- Tabular Datasets: Jannis and CoverType.
Key Insights and Ideas
- GPU Under-utilization is Key: The primary bottleneck in AT isn't always the computation itself, but rather the fixed overhead associated with launching the attack function, especially on powerful GPUs with smaller models.
- Overhead Throughput ($O^*$): This novel metric quantifies the maximum amount of data a GPU can process within its fixed overhead time during attack generation. It's a crucial indicator of potential speedup.
- "Hitchhiking" for Speed: By packing multiple minibatches into a single attack generation call (up to $O^*$), AttackRider effectively amortizes the overhead cost, leading to significant speedups.
- Shuffle and Divide for Robustness: The proposed shuffle and divide strategy for model updates is critical for preserving model robustness and generalization, preventing issues associated with larger effective batch sizes.
- Targeted Application: AttackRider is most effective when GPUs are powerful relative to model size, or when the base AT uses small minibatches, leading to a large $O^*$.
Key Results
- AttackRider achieved significant speedups across various base AT algorithms and datasets.
- On CIFAR-10 with the RN model, applying AttackRider to BulletTrain resulted in a speedup of 3.45x, and to FT-T on Jannis dataset yielded 3.77x speedup, while maintaining comparable robustness.
- The paper demonstrates that for base ATs like BulletTrain, choosing a small packing size ($e \in \{1,2\}$) offers a good balance between speedup and accuracy. For other base ATs, setting $e = O^*$ provides the best speedup.
- Ablation studies showed that the Shuffle and Divide functions are essential for maintaining clean and robust accuracy, with significant drops observed when they were disabled.
Limitations and Future Work
- Overestimation of Speedup: The estimated speedup ($rSpd$) can be higher than the actual wall-clock speedup ($Spd$) due to approximations in estimating the ratio of attack time to update time.
- Outdated Adversarial Examples: While the paper argues that model robustness can be preserved with moderately outdated adversarial examples, a larger packing size ($e$) can lead to more outdated examples, potentially impacting accuracy. The authors suggest that a balance needs to be struck.
- Choosing $e$: While guidelines are provided (e.g., $e \in \{1,2\}$ for BulletTrain, $e=O^*$ for others), finding the optimal $e$ might still require some hyperparameter tuning.
- Future Work: The paper implicitly suggests exploring more sophisticated methods for selecting influential examples or adapting the packing strategy dynamically based on GPU utilization and model state. Further investigation into the theoretical bounds of robustness with outdated adversarial examples could also be beneficial.
Summary updated