S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights

Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, Xiaowei Huang

TL;DR

This paper introduces Second-Order Statistics Optimization ($S^2O$), a novel approach that enhances adversarial training by treating neural network weights as correlated random variables rather than independent entities. By relaxing the traditional PAC-Bayesian assumption of weight independence, the authors derive a tighter robust generalization bound and provide a practical, efficient regularizer that significantly boosts both model robustness and clean accuracy.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples—subtle, human-imperceptible perturbations designed to fool models into making confident, incorrect predictions. While adversarial training (AT) is the most effective defense, it often suffers from a frustrating trade-off: improving a model's robustness against attacks usually degrades its accuracy on clean, unperturbed data.

To understand this mathematically, researchers use the PAC-Bayesian framework to derive theoretical bounds on a model's "robust generalization error." However, existing frameworks rely on a highly unrealistic assumption: that the weights of a neural network are statistically independent. This mismatch between theory and reality leads to loose generalization bounds that fail to accurately guide the training process. This paper bridges this gap by explicitly modeling the statistical correlations between weights.

Methods

The authors propose a two-step solution encompassing a new theoretical framework and a practical optimization algorithm:

  1. Relaxing Weight Independence: The authors redefine the random perturbations applied to weights within the PAC-Bayesian framework as a non-spherical Gaussian distribution. This allows them to explicitly incorporate second-order statistics—specifically, weight correlation and normalized covariance matrices—into the robust generalization bound.
  2. The $S^2O$ Regularizer: Directly optimizing the full weight correlation matrix is computationally impossible for deep networks. To make this practical, the authors leverage a Kronecker-factored Laplace approximation (inspired by second-order optimization techniques). This mathematical shortcut allows them to control weight correlation by monitoring and minimizing the Frobenius norm of the normalized covariance of the network's post-activations ($\|A\|_F^2$).

To maximize efficiency, the $S^2O$ regularizer is applied only to the final layer of the network during adversarial training.

The $S^2O$ Optimization Framework

Datasets

The effectiveness of $S^2O$ was evaluated across several standard computer vision benchmarks:

  • CIFAR-10 & CIFAR-100: Standard 32x32 image classification datasets.
  • SVHN (Street View House Numbers): Real-world cropped digit images.
  • Tiny-ImageNet: A downscaled subset of ImageNet containing 200 classes.
  • Imagenette: A smaller, 10-class subset of ImageNet used for rapid prototyping.

Key Insights and Ideas

  • Correlations Drive Robustness: The core "aha moment" is that minimizing the spectral norm while maximizing the determinant of weight correlation matrices directly tightens the robust generalization bound.
  • Post-Activations as a Proxy: Instead of wrestling with massive weight-by-weight Hessian matrices, the authors realized they could look at the covariance of the layer activations. This elegant approximation makes second-order weight optimization computationally feasible.
  • Synergistic Effects: $S^2O$ isn't just a standalone defense; it acts as a force multiplier. Because it alters how the weights behave statistically, it can be seamlessly combined with existing state-of-the-art adversarial training methods like TRADES or Adversarial Weight Perturbation (AWP).
Fig. 4. We train PreAct ResNet18 with AT and AT+S2O on CIFAR-10, and show the results of partial weights. The results estimated from adversarial data are presented in the top figures, while the bottom figures show the results estimated from clean data. (a) shows the normalized spectral norm of Rc, Rr, and the determinant of R, with sampling estimation and Laplace approximation respectively. (b) and (c) demonstrate the absolute correlation matrix of partial weights, for AT and AT+S2O respectively.

Key Results

The empirical results demonstrate that $S^2O$ consistently improves both clean accuracy and robust accuracy across multiple network architectures (including PreAct ResNet-18, WideResNet, and Vision Transformers like ViT-B and DeiT-S).

  • Vanilla AT Boost: On CIFAR-10 using a PreAct ResNet-18, adding $S^2O$ to standard adversarial training improved clean accuracy from 82.41% to 83.65% and robust accuracy (under a PGD-20 attack) from 52.77% to 55.11%.
  • Enhancing State-of-the-Art: When integrated with TRADES and AWP on a WideResNet architecture, the combined model achieved top-tier performance, pushing AutoAttack (AA) robust accuracy to 55.9% on CIFAR-10.
  • Mitigating the Trade-off: Unlike standard defenses that sacrifice clean accuracy for robustness, $S^2O$ frequently improved both metrics simultaneously, effectively softening the robustness-accuracy trade-off curve.
Performance of $S^2O$ combined with TRADES and AWP

Limitations and Future Work

While highly effective, $S^2O$ introduces a modest computational overhead, increasing training time by roughly 20% per epoch due to the calculation of the post-activation correlation matrices for both clean and adversarial data. The authors noted that this overhead can be cut in half by calculating correlations exclusively on adversarial data, though this results in a slight drop in final model performance.

Additionally, for the sake of computational efficiency, the current implementation only applies the regularizer to the network's final layer. Exploring efficient ways to scale $S^2O$ across all layers of deep architectures remains an open direction for future research.

Summary updated

Abstract

Adversarial training has emerged as a highly effective way to improve the robustness of deep neural networks (DNNs). It is typically conceptualized as a min-max optimization problem over model weights and adversarial perturbations, where the weights are optimized using gradient descent methods, such as SGD. In this paper, we propose a novel approach by treating model weights as random variables, which paves the way for enhancing adversarial training through \textbf{S}econd-Order \textbf{S}tatistics \textbf{O}ptimization (S$^2$O) over model weights. We challenge and relax a prevalent, yet often unrealistic, assumption in prior PAC-Bayesian frameworks: the statistical independence of weights. From this relaxation, we derive an improved PAC-Bayesian robust generalization bound. Our theoretical developments suggest that optimizing the second-order statistics of weights can substantially tighten this bound. We complement this theoretical insight by conducting an extensive set of experiments that demonstrate that S$^2$O not only enhances the robustness and generalization of neural networks when used in isolation, but also seamlessly augments other state-of-the-art adversarial training techniques. The code is available at https://github.com/Alexkael/S2O.
BibTeX
@article{jin2025s,
  author = {Gaojie Jin and Xinping Yi and Wei Huang and Sven Schewe and Xiaowei Huang},
  title = {{S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3577384},
}

Loading related content...

Figure