Parameter Interpolation Adversarial Training for Robust Image Classification

Xin Liu, Yichen Yang, Kun He, John E. Hopcroft

Introduction

Adversarial Training (AT) remains the most effective defense against adversarial attacks, which involve crafting small, imperceptible perturbations to fool deep neural networks. However, the optimization process of AT is notoriously unstable. Researchers have identified two significant pathologies: robust oscillations, where accuracy against attacks fluctuates wildly in early training, and robust overfitting, where the model's generalization to unseen adversarial data degrades even as training accuracy improves.

This paper proposes that these issues stem from abrupt shifts in the model's decision boundary between training epochs. To mitigate this, the authors introduce Parameter Interpolation Adversarial Training (PIAT) and a Normalized Mean Square Error (NMSE) loss, providing a smoother optimization path for robust image classification.

Proposed Methods

Parameter Interpolation Adversarial Training (PIAT)

The core innovation of PIAT is a temporal smoothing mechanism applied to the model parameters. Instead of using the weights from the end of an epoch directly for the next, PIAT computes a weighted average between the previous epoch's parameters ($\theta'_{t-1}$) and the current epoch's parameters ($\theta_t$):

$$\theta'_t = \lambda \cdot \theta'_{t-1} + (1 - \lambda) \cdot \theta_t$$

A key feature of PIAT is the dynamic interpolation factor ($\lambda$). In the early stages of training, $\lambda$ is kept small to allow the model to learn quickly from the data. As training progresses, $\lambda$ increases toward 1. This shift places more weight on the accumulated knowledge of previous epochs, preventing the decision boundary from making erratic "jumps" and effectively suppressing robust overfitting in the final stages of training.

Normalized Mean Square Error (NMSE)

To further stabilize training, the authors propose NMSE for logit alignment. Traditional methods like Adversarial Logit Pairing (ALP) attempt to match the raw logit outputs of clean and adversarial examples. However, this can be overly restrictive and harm standard accuracy. NMSE addresses this by normalizing the logits using the $l_2$-norm before calculating the mean square error. This approach focuses on the relative distribution of class scores rather than their absolute magnitudes, aligning the model's decision logic more effectively without sacrificing clean performance.

Experimental Results

The PIAT framework was evaluated across several benchmarks, including CIFAR10, CIFAR100, and SVHN, using both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs).

Robustness and Stability

On the CIFAR10 dataset using a ResNet18 architecture, the combination of PIAT and NMSE achieved a robust accuracy of 48.80% under the rigorous AutoAttack (AA) benchmark. This performance surpasses established methods such as TRADES (48.04%) and MART (47.45%).

Furthermore, PIAT demonstrated superior stability. The "Difference" (Diff) metric—the gap between the best robust accuracy achieved during training and the final accuracy—was significantly reduced. For instance, on CIFAR10 with PGD20 attacks, PIAT reduced the Diff to 0.93%, compared to 4.43% for standard PGD-AT, indicating a successful mitigation of robust overfitting.

Performance on Vision Transformers

PIAT is architecture-agnostic and provides consistent improvements for modern architectures. When integrated with existing defenses for Vision Transformers (such as A&P), PIAT improved the robust accuracy of ConViT-B on CIFAR10 from 49.01% to 51.86%.

Conclusion

The PIAT framework offers a robust solution to the optimization challenges inherent in adversarial training. By smoothing the transition of parameters between epochs and focusing on relative logit alignment via NMSE, the method achieves state-of-the-art robustness while maintaining stability. Future work may extend this interpolation strategy to other unstable training regimes, such as Reinforcement Learning or Generative Adversarial Networks (GANs).

Summary updated

BibTeX
@article{liu2025parameter,
  author = {Xin Liu and Yichen Yang and Kun He and John E. Hopcroft},
  title = {{Parameter Interpolation Adversarial Training for Robust Image Classification}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2025},
  doi = {10.1109/tifs.2025.3533925},
}
Figure