How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning?

Wenjun Ding, Ying An, Lixing Chen, Shichao Kan, Fan Wu, Zhe Qu

Introduction

Federated Learning (FL) enables multiple parties to collaboratively train models without sharing raw data, but it remains highly vulnerable to adversarial attacks. Federated Adversarial Learning (FAL) was developed to build robustness in these distributed settings. However, FAL models often face a significant "generalization gap," where they perform well on training data but fail on unseen test data. Analyzing this gap is challenging because adversarial loss functions are non-smooth—jagged and spiky—which breaks standard mathematical tools used to measure algorithmic stability.

Smoothness Approximation Methods

To address the non-smoothness of adversarial objectives, researchers use approximation techniques to create a more tractable optimization landscape. This paper provides a rigorous study of three such methods:

  1. Surrogate Smoothness Approximation (SSA): Replaces the jagged loss with a smoother surrogate function.
  2. Randomized Smoothness Approximation (RSA): Adds random noise to model parameters during training to "blur" the loss surface.
  3. Over-Parameterized Smoothness Approximation (OPSA): Utilizes very wide neural networks, which naturally possess smoother landscapes.

These methods are evaluated using two algorithmic frameworks: Vanilla FAL (VFAL) and Slack FAL (SFAL). SFAL dynamically reweights client contributions based on local adversarial loss to mitigate the negative effects of data heterogeneity.

Comparative Analysis of Approximation Methods

The study identifies Randomized Smoothness Approximation (RSA) as the most effective tool for reducing generalization error. By intentionally flattening the weight space with noise, RSA prevents the model from falling into sharp minima that lead to poor generalization.

Comparison of generalization gaps across SSA, RSA, and OPSA under varying data heterogeneity ($a$).

As illustrated in Figure 3, data heterogeneity (represented by the skew parameter $a$) is a primary driver of generalization error. Lower values of $a$ (higher heterogeneity) consistently lead to larger gaps. Across all approximation methods, the SFAL algorithm (represented by the lower curves in each pair) demonstrates superior stability compared to VFAL, effectively "cleaning up" the noise introduced by non-IID data distributions.

Key Results and Scaling Laws

Theoretical analysis shows that RSA achieves a generalization error scaling of $O(T^{1/4})$, significantly better than the $O(T)$ scaling of SSA. Empirical validation on the SVHN dataset confirms these findings, with RSA maintaining the smallest generalization gap.

Impact of the number of clients ($m$) on the generalization gap in the OPSA setting.

Furthermore, the research highlights the benefit of client scaling. As shown in Figure 2, increasing the number of clients ($m$) helps disperse the negative effects of heterogeneity, effectively diluting the generalization error. However, the authors also warn of the "curse of width" in over-parameterized models (OPSA), where generalization error can increase with the square root of the network width if not properly managed.

Conclusion

This work provides the first deep dive into how smoothness approximations impact generalization in Federated Adversarial Learning. While RSA offers the best theoretical and empirical performance, its computational cost remains a hurdle. Future research will focus on developing more efficient, stability-aware aggregation rules to further bridge the generalization gap in complex, heterogeneous environments.

Summary updated

Abstract

Federated Adversarial Learning (FAL) is a robust framework for resisting adversarial attacks on federated learning. Although some FAL studies have developed efficient algorithms, they primarily focus on convergence performance and overlook generalization. Generalization is crucial for evaluating algorithm performance on unseen data. However, generalization analysis is more challenging due to non-smooth adversarial loss functions. A common approach to addressing this issue is to leverage smoothness approximation. In this paper, we develop algorithm stability measures to evaluate the generalization performance of two popular FAL algorithms: Vanilla FAL (VFAL) and Slack FAL (SFAL), using three different smooth approximation methods: 1) Surrogate Smoothness Approximation (SSA), (2) Randomized Smoothness Approximation (RSA), and (3) Over-Parameterized Smoothness Approximation (OPSA). Based on our in-depth analysis, we answer how to properly set the smoothness approximation method to mitigate generalization error in FAL. Moreover, we identify RSA as the most effective generalization error reduction method. In highly data-heterogeneous scenarios, we also recommend employing SFAL to mitigate the deterioration of generalization performance caused by heterogeneity. Based on our theoretical results, we provide insights to help develop more efficient FAL algorithms, such as designing new metrics and dynamic aggregation rules to mitigate heterogeneity.
BibTeX
@inproceedings{ding2025how,
  author = {Wenjun Ding and Ying An and Lixing Chen and Shichao Kan and Fan Wu and Zhe Qu},
  title = {{How Does the Smoothness Approximation Method Facilitate Generalization for Federated Adversarial Learning?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33788},
}
Figure