Guided Interpolation for Adversarial Training

Chen Chen, Jingfeng Zhang, Xilie Xu, Tianlei Hu, Gang Niu, Masashi SugiyamaarXiv 2021 · arXiv

Abstract

Adversarial Training (AT) is the primary defense against adversarial attacks, yet its effectiveness often plateaus as training progresses. This stagnation occurs because the model becomes proficient at "guarding" its training data, reducing the pool of challenging samples available for learning. We present the Guided Interpolation Framework (GIF), a method that utilizes meta-information from previous training epochs to selectively interpolate "attackable" data. Unlike standard Mixup, GIF employs a fixed interpolation weight to preserve local invariance and sharp decision boundaries. Our results on CIFAR-10 and SVHN demonstrate that GIF significantly enhances robust accuracy and mitigates the common issue of robust overfitting.

The Challenge of "Guarded" Data in Adversarial Training

While Adversarial Training (AT) is the gold standard for improving model robustness, it suffers from diminishing returns. As a model's robustness increases, the training data becomes increasingly "guarded"—it becomes harder to find adversarial perturbations within a defined epsilon-ball that can successfully fool the model.

Ratio of attackable data vs. robust test accuracy over training epochs.

As shown in the figure above, there is a clear correlation between the ratio of attackable data and the model's robust test accuracy. When the availability of attackable samples drops, the improvement in robustness stalls. While data augmentation techniques like Mixup can generate more samples, they often introduce undesirable linear behavior between classes, which conflicts with the goal of "local invariance"—the principle that a model's prediction should remain constant within a small neighborhood of a data point.

The Guided Interpolation Framework (GIF)

To address the plateau in robustness without sacrificing local invariance, we propose the Guided Interpolation Framework (GIF). GIF is designed to provide a continuous stream of high-quality, attackable data throughout the training process.

Key Components

  1. Guided Selection: GIF identifies "attackable" samples from the previous epoch—those where an adversarial attack successfully flipped the prediction—and prioritizes them for interpolation in the current epoch.
  2. Fixed Interpolation Weight: Standard Mixup uses a random weight ($\lambda$) to blend images, creating smooth transitions. GIF fixes $\lambda$ at 0.5. This 50/50 split encourages the model to maintain sharper decision boundaries, which is crucial for adversarial robustness.
  3. Burn-in Period: The framework begins with a standard AT "burn-in" phase (e.g., 30 epochs) to establish a baseline before introducing interpolated samples.
Comparison of decision boundaries and prediction curves for AT, AT-mixup, and AT-GIF.

As visualized in the decision boundary plots, AT-mixup creates wide, linear transition zones between classes. In contrast, AT-GIF maintains the sharp, invariant boundaries seen in standard AT while successfully incorporating interpolated data to challenge the model.

Experimental Results

We evaluated GIF on CIFAR-10 and SVHN benchmarks using ResNet-18 and Wide ResNet-32-10 architectures. GIF consistently outperformed both standard AT and Mixup-based variants.

  • Robust Accuracy: On CIFAR-10 with ResNet-18, AT-GIF achieved 53.57% robust accuracy under PGD-20 attacks, surpassing the 51.05% achieved by standard AT.
  • Integration with SOTA: GIF is a flexible framework. When integrated with GAIRAT on Wide ResNet-32-10, it boosted robust test accuracy from 56.69% to 64.97%, an improvement of over 8%.
  • Mitigating Overfitting: Standard AT often experiences "robust overfitting," where performance declines in the late stages of training. By providing a diverse and challenging set of attackable samples throughout training, GIF helps maintain and even improve performance in later epochs.

Conclusion

The Guided Interpolation Framework (GIF) offers a principled way to overcome the diminishing returns of Adversarial Training. By leveraging meta-information to select attackable data and using fixed-weight interpolation to preserve local invariance, GIF provides a robust boost to model performance across various architectures and existing AT methods.

Summary updated

Abstract

To enhance adversarial robustness, adversarial training learns deep neural networks on the adversarial variants generated by their natural data. However, as the training progresses, the training data becomes less and less attackable, undermining the robustness enhancement. A straightforward remedy is to incorporate more training data, but sometimes incurring an unaffordable cost. In this paper, to mitigate this issue, we propose the guided interpolation framework (GIF): in each epoch, the GIF employs the previous epoch's meta information to guide the data's interpolation. Compared with the vanilla mixup, the GIF can provide a higher ratio of attackable data, which is beneficial to the robustness enhancement; it meanwhile mitigates the model's linear behavior between classes, where the linear behavior is favorable to generalization but not to the robustness. As a result, the GIF encourages the model to predict invariantly in the cluster of each class. Experiments demonstrate that the GIF can indeed enhance adversarial robustness on various adversarial training methods and various datasets.
BibTeX
@misc{chen2021guided,
  author = {Chen Chen and Jingfeng Zhang and Xilie Xu and Tianlei Hu and Gang Niu and Gang Chen and Masashi Sugiyama},
  title = {{Guided Interpolation for Adversarial Training}},
  year = {2021},
  eprint = {2102.07327},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.07327},
}
Figure