Defending Against Sophisticated Poisoning Attacks with RL-based Aggregation in Federated Learning

Yujing Wang, Hainan Zhang, Sijia Wen, Wangjie Qiu, Binghui Guo

TL;DR

Federated learning is highly vulnerable to sophisticated model poisoning attacks that easily bypass static, statistics-based defenses. The authors introduce AdaAggRL, an adaptive defense framework that uses gradient inversion to reconstruct proxy client data distributions and leverages reinforcement learning to dynamically weight or exclude clients based on their distribution stability over time.

What Problem Does This Paper Address?

Federated Learning (FL) allows decentralized devices to collaboratively train a shared global model while keeping their raw data localized and private. However, this decentralized nature leaves FL wide open to model poisoning attacks, where malicious clients upload manipulated gradients designed to degrade the global model's accuracy.

Traditional defenses typically rely on static statistical heuristics (such as gradient clipping or taking the coordinate-wise median) or require the server to hold a clean "root" dataset for validation. While these methods work against simple, myopic attacks, they fail miserably against sophisticated, learnable attacks (like RL-based attacks). These advanced attacks optimize for statistical stealth, effectively mimicking benign updates while still corrupting the global model. This paper addresses the critical need for an adaptive defense capable of thwarting these dynamic, server-tailored poisoning strategies.

Methods

The proposed framework, AdaAggRL (Adaptive Aggregation via Reinforcement Learning), operates at the server level during the aggregation phase using three main steps:

  1. Distribution Learning: For each training round, the server applies the Inverting Gradients (IG) method to reconstruct a batch of dummy data that matches the gradients uploaded by each client. This acts as a proxy for the client's actual training data distribution.
  2. Environmental Cues (Similarity Metrics): The server passes the reconstructed data through a pre-trained CNN to extract feature vectors. It then uses Maximum Mean Discrepancy (MMD) to calculate three pairwise similarity metrics:
  3. Current client distribution vs. its historical distribution ($S_{k,cl}$)
  4. Current client distribution vs. the global model distribution ($S_{k,cg}$)
  5. Historical client distribution vs. the global model distribution ($S_{k,lg}$) It also calculates a reconstruction similarity ($S_{k,R}$) to measure the quality and trustworthiness of the gradient inversion process itself.

  6. Policy Learning: The server feeds these four metrics into a Twin Delayed DDPG (TD3) reinforcement learning agent. The RL agent adaptively determines the optimal aggregation weights for each client and sets a filtering threshold ($\delta$) to completely exclude clients displaying highly irregular distribution shifts.

Figure 3: An overview of our AdaAggRL

Datasets

The authors evaluated AdaAggRL across both Computer Vision (CV) and Natural Language Processing (NLP) tasks using the following datasets:

  • MNIST, F-MNIST, and EMNIST: Used to evaluate image classification performance under varying degrees of non-i.i.d. (non-identically and independently distributed) data splits.
  • CIFAR-10: Paired with a ResNet18 architecture to test the defense on a more complex CV task.
  • IMDB: A movie review dataset paired with a TextCNN framework to verify that the distribution stability principles hold true for NLP embedding spaces.

Key Insights and Ideas

  • Distribution Stability as a Signature: The core "aha!" moment of the paper is that benign clients naturally exhibit high data distribution stability across training rounds because they sample randomly from a fixed local dataset. Malicious clients, by contrast, must continuously alter their gradients to manipulate the global model, resulting in erratic, unstable simulated data distributions over time.
  • Global Alignment: Because the global model represents the average state of normal data, benign client distributions consistently align closely with it. Malicious clients inherently deviate from this global average.
  • Stealth Attacks Leave Footprints: Even when an advanced attack successfully tricks median- or clipping-based defenses by keeping its gradients statistically inconspicuous, it cannot hide the fundamental instability it introduces to the underlying data distribution.

Key Results

AdaAggRL demonstrated remarkable robustness, consistently outperforming baselines like Krum, Median, C-Median, FLtrust, Clipping, Feddefender, and FedVal across all datasets.

  • Defeating Sophisticated Attacks: Against the highly deceptive RL-attack on the MNIST dataset, traditional defenses like C-Median and Krum saw global model accuracies plunge to 42.5% and 15.6%, respectively. AdaAggRL maintained a stable global accuracy of 95.5%.
  • High Attack Tolerance: As shown in the paper's analysis, AdaAggRL can tolerate a network containing up to 90% malicious clients with only a minor drop in performance, whereas standard baselines break down completely once malicious clients exceed 30%.
  • Robustness to Non-i.i.d. Data: Even under extreme data heterogeneity, the RL agent successfully learned to adaptively lower the weights of global model-related similarity scores to prevent false positives among benign clients.

Limitations and Future Work

  • Computational Complexity: Reconstructing data distributions via gradient inversion is computationally expensive. The authors note that AdaAggRL increases training time per FL round by an average of 21.4% on MNIST and 50.1% on CIFAR-10 compared to standard baselines.
  • Arms Race of Attack vs. Defense: The authors acknowledge that a clever attacker aware of AdaAggRL could potentially formulate a new optimization objective designed to explicitly stabilize their simulated data distribution changes while still poisoning the model. Developing proactive defenses against such "distribution-stabilized" attacks remains an open avenue for future research.

Summary updated

Abstract

Federated learning is highly susceptible to model poisoning attacks, especially those meticulously crafted for servers. Traditional defense methods mainly focus on updating assessments or robust aggregation against manually crafted myopic attacks. When facing advanced attacks, their defense stability is notably insufficient. Therefore, it is imperative to develop adaptive defenses against such advanced poisoning attacks. We find that benign clients exhibit significantly higher data distribution stability than malicious clients in federated learning in both CV and NLP tasks. Therefore, the malicious clients can be recognized by observing the stability of their data distribution. In this paper, we propose AdaAggRL, an RL-based Adaptive Aggregation method, to defend against sophisticated poisoning attacks. Specifically, we first utilize distribution learning to simulate the clients' data distributions. Then, we use the maximum mean discrepancy (MMD) to calculate the pairwise similarity of the current local model data distribution, its historical data distribution, and global model data distribution. Finally, we use policy learning to adaptively determine the aggregation weights based on the above similarities. Experiments on four real-world datasets demonstrate that the proposed defense model significantly outperforms widely adopted defense models for sophisticated attacks.
BibTeX
@inproceedings{wang2025defending,
  author = {Yujing Wang and Hainan Zhang and Sijia Wen and Wangjie Qiu and Binghui Guo},
  title = {{Defending Against Sophisticated Poisoning Attacks with RL-based Aggregation in Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=PXx4F0YfoS},
}
Figure