Abstract
Federated learning is highly susceptible to model poisoning attacks, especially those meticulously crafted for servers. Traditional defense methods mainly focus on updating assessments or robust aggregation against manually crafted myopic attacks. When facing advanced attacks, their defense stability is notably insufficient. Therefore, it is imperative to develop adaptive defenses against such advanced poisoning attacks. We find that benign clients exhibit significantly higher data distribution stability than malicious clients in federated learning in both CV and NLP tasks. Therefore, the malicious clients can be recognized by observing the stability of their data distribution. In this paper, we propose AdaAggRL, an RL-based Adaptive Aggregation method, to defend against sophisticated poisoning attacks. Specifically, we first utilize distribution learning to simulate the clients' data distributions. Then, we use the maximum mean discrepancy (MMD) to calculate the pairwise similarity of the current local model data distribution, its historical data distribution, and global model data distribution. Finally, we use policy learning to adaptively determine the aggregation weights based on the above similarities. Experiments on four real-world datasets demonstrate that the proposed defense model significantly outperforms widely adopted defense models for sophisticated attacks.
Introduction
Federated Learning (FL) has emerged as a powerful paradigm for training machine learning models on decentralized data while preserving user privacy. However, this decentralized nature introduces a critical vulnerability: the system is highly susceptible to "poisoning attacks." In these scenarios, malicious clients send corrupted gradient updates to the central server, aiming to degrade the global model's performance or install backdoors.
While traditional defenses—such as taking the coordinate-wise median of updates or employing gradient clipping—can mitigate simple, "myopic" attacks, they often fail against sophisticated, learnable attacks. Advanced attackers can utilize Reinforcement Learning (RL) to craft updates that appear statistically normal to the server but are strategically designed to be destructive. This paper introduces AdaAggRL, a novel defense mechanism that shifts the focus from raw parameter values to the underlying data distribution stability represented by those parameters.
The AdaAggRL Framework
AdaAggRL (RL-based Adaptive Aggregation) is built on the observation that benign clients typically possess stable data distributions over time, whereas malicious clients must frequently shift their gradients to remain effective, resulting in erratic distributional signatures.
The framework operates in three distinct stages:
- Distribution Learning: To audit clients without accessing raw data, the server employs an "Inverting Gradients" (IG) technique. By analyzing a client’s model update, the server reconstructs "dummy" data that captures the statistical "fingerprint" of the client's local dataset.
- Environmental Cues (Similarity Metrics): The server calculates the Maximum Mean Discrepancy (MMD) to compare three key relationships: the client’s current distribution vs. its historical distribution, the current distribution vs. the global model’s distribution, and the historical distribution vs. the global distribution.
- RL-based Policy Learning: Using the TD3 (Twin Delayed DDPG) algorithm, the server learns a policy to assign aggregation weights. If a client's distribution is erratic or deviates significantly from the global norm, the RL agent lowers its weight. A "penalty vector" is also maintained to track and penalize repeated malicious behavior across multiple rounds.
Analysis of Distributional Stability
A core insight of this work is that distributional stability serves as a reliable proxy for trust. Benign clients exhibit consistent data distributions, while malicious actors leave a detectable trail in the "distribution space" that is far harder to mask than anomalies in the "parameter space."
As shown in the figure above, across various attack types including the sophisticated RL-attack, the similarity metrics for benign clients remain high and stable. In contrast, malicious clients show significant fluctuations and lower overall similarity, providing a clear signal for the RL-based aggregator to reduce their influence.
Experimental Evaluation
The authors evaluated AdaAggRL across Computer Vision (MNIST, Fashion-MNIST, EMNIST, CIFAR-10) and Natural Language Processing (IMDB review dataset) tasks. They specifically focused on non-I.I.D. settings, where data is heterogeneously distributed across clients, representing a common real-world challenge.
Key Results
- Superior Robustness: Against state-of-the-art learnable RL-attacks, AdaAggRL maintained approximately 96% accuracy on MNIST. In comparison, traditional defenses like Krum or Coordinate-Median saw performance plummet to between 15% and 68%.
- High Attack Tolerance: The system demonstrated remarkable resilience, maintaining stable global accuracy even when up to 90% of the participating clients were malicious. Most baseline defenses failed once malicious participation exceeded 30%.
- Domain Generalization: The method proved effective across different architectures (CNNs and ResNet18) and diverse data domains (Images and Text).
- Efficiency: While gradient inversion adds computational complexity, the authors found that AdaAggRL only increases training time by 21% to 50% per round—a justifiable trade-off for the significant security gains.
Discussion and Future Directions
While AdaAggRL provides a robust defense, it is not without limitations. The gradient inversion process requires multiple optimization steps, which may pose scalability challenges for extremely large FL deployments. Furthermore, the security landscape is an ongoing "arms race"; future attackers may attempt to develop "distribution-aware" strategies designed to mimic the stability of benign clients. Future work will likely focus on optimizing the computational efficiency of the distribution learning phase and exploring even more resilient similarity metrics.
Summary updated