Abstract
Federated learning has exhibited vulnerabilities to Byzantine attacks, where the Byzantine attackers can send arbitrary gradients to a central server to destroy the convergence and performance of the global model. A wealth of robust AGgregation Rules (AGRs) have been proposed to defend against Byzantine attacks. However, Byzantine clients can still circumvent robust AGRs when data is non-Identically and Independently Distributed (non-IID). In this paper, we first reveal the root causes of performance degradation of current robust AGRs in non-IID settings: the curse of dimensionality and gradient heterogeneity. In order to address this issue, we propose GAS, a gradient splitting approach that can successfully adapt existing robust AGRs to non-IID settings. We also provide a detailed convergence analysis when the existing robust AGRs are combined with GAS. Experiments on various real-world datasets verify the efficacy of our proposed GAS.
Introduction
Federated Learning (FL) enables collaborative model training without centralizing raw data, but this decentralization introduces significant security risks. Specifically, Byzantine attacks—where malicious participants submit poisoned gradients—can easily derail model convergence. While various Robust Aggregation Rules (AGRs) like Multi-Krum, Bulyan, or Median have been developed to mitigate these threats, they often falter in real-world non-IID (heterogeneous) environments. In such settings, honest gradients naturally vary, making it difficult for standard filters to distinguish between benign diversity and malicious poisoning.
This paper identifies two primary obstacles to robust aggregation in non-IID settings: the curse of dimensionality, which provides attackers with ample space to hide malicious updates, and gradient heterogeneity, which causes filters to inadvertently exclude valid updates. To overcome these, we introduce GAS (GrAdient Splitting), a framework that enhances existing AGRs by operating in lower-dimensional subspaces.
The GAS Framework
GAS is a "plug-and-play" enhancement that can be integrated with most existing robust AGRs. The process follows three main steps:
- Splitting: The central server randomly partitions high-dimensional gradient vectors into $p$ smaller, low-dimensional sub-vectors.
- Identification: A standard robust AGR (e.g., Median, Multi-Krum) is applied to each group of sub-vectors independently to establish a reference for honest behavior. Each client receives an "abnormal score" based on the deviation of their sub-vectors from these references.
- Aggregation: The server identifies clients with the lowest cumulative abnormal scores—representing the most likely honest participants—and averages their original, full-dimensional gradients to update the global model.
By decomposing the problem, GAS significantly reduces the statistical "room" available for attackers to craft deceptive gradients that appear honest across all split segments.
Experimental Evaluation
We evaluated GAS using four real-world datasets (CIFAR-10, CIFAR-100, ImageNet-12, and FEMNIST) with simulated non-IID partitions using Dirichlet distributions. We tested against several attack types, including BitFlip, LabelFlip, and LIE.
Key Results and Insights
The integration of GAS consistently improved the performance of all tested AGRs. For instance, in scenarios where standard defenses like Bulyan or RFA struggled due to high data heterogeneity, GAS restored high levels of accuracy.
As shown in Figure 5, the Bulyan aggregator's performance is significantly stabilized and improved when paired with GAS, particularly in challenging attack scenarios where the baseline (solid blue line) fails to converge effectively. Similarly, Figure 6 demonstrates the gains for the RFA aggregator.
Beyond raw accuracy, GAS-enhanced models exhibited lower gradient deviation from the optimal honest path. In highly non-IID settings ($\beta=0.3$), GAS maintained functional accuracy (e.g., ~52% on CIFAR-10) while standalone defenses like Multi-Krum often collapsed to near-random performance levels.
Conclusion
GAS addresses the fundamental weaknesses of current robust aggregation rules in heterogeneous data environments by leveraging gradient decomposition. While future work may explore more sophisticated splitting strategies (e.g., layer-wise splitting) and adaptive attack resistance, GAS currently offers a computationally efficient ($O(d)$) and highly effective solution for securing Federated Learning systems. The implementation is available at https://github.com/YuchenLiu-a/byzantine-gas.
Summary updated