Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework

Henger Li, Xiaolin Sun, Zizhan ZhengNeurIPS 2022 · OpenReview

Overview

Federated Learning (FL) allows multiple devices to train a shared model without ever sending their raw, private data to a central server. While privacy-preserving, FL is notoriously vulnerable to poisoning attacks, where malicious participants send "dirty" model updates to degrade the global model's accuracy.

To counter this, researchers have developed robust aggregation rules (like Krum or Trimmed Mean) designed to filter out outliers. However, most existing attacks are myopic—they try to do maximum damage in a single round without considering the long-term trajectory of the model. Current defenses often rely on the assumption that attackers lack long-term strategic planning.

This paper introduces a sophisticated "model-based" reinforcement learning (RL) framework that automatically learns how to sabotage FL systems over time. By using gradient inversion to "spy" on the global data distribution and training an RL agent in a simulated environment, this attack successfully bypasses state-of-the-art robust defenses.

The Attack Framework

The authors propose a three-phase framework that treats attacking FL as a Markov Decision Process (MDP).

The proposed model-based reinforcement learning attack framework, illustrating Distribution Learning and Policy Learning phases.

1. Distribution Learning (Building the "World Model")

Since the attacker doesn't have access to the benign clients' private data, they use Gradient Inversion. By analyzing the difference between global model updates, they reconstruct a "synthetic" version of the aggregate data distribution. To make their simulator more accurate, they employ a denoising auto-encoder to clean up these reconstructed images.

Examples of noisy images reconstructed via gradient inversion and their denoised counterparts used to build the simulator.

2. Policy Learning (The Sandbox)

The attacker builds a simulator of the FL environment using the learned data distribution. They then train an RL agent (using algorithms like TD3 or PPO) in this "sandbox." * State Space: Instead of using the massive weight vector of a neural network, the state is compressed to the parameters of the last hidden layer. * Action Space: Rather than picking raw weights, the RL agent learns to tune three high-level parameters ($\gamma, E, \lambda$) that guide a gradient ascent search. This finds the most damaging update direction that still "looks" enough like a normal update to bypass filters.

3. Attack Execution

Once the policy is trained, the malicious nodes apply it during the real FL training process, adapting their poisoned updates based on the learned strategy.

Experimental Setup

The framework was evaluated on four standard computer vision benchmarks: MNIST (handwritten digits), Fashion-MNIST (clothing items), EMNIST (letters and digits), and CIFAR-10 (color images). The authors tested these in both i.i.d. (evenly distributed data) and non-i.i.d. (heterogeneous data across clients) settings to reflect real-world FL complexities.

Key Insights

  • From Myopic to Strategic: Most poisoning attacks are "greedy." This RL approach is strategic; it might choose a subtler attack now to ensure it isn't kicked out of the system, allowing it to inflict more damage later.
  • Gradient Inversion as a Weapon: Usually discussed as a privacy risk (leaking user data), this paper shows gradient inversion is also a security risk. It allows an attacker to build a highly accurate simulator of the system they want to subvert.
  • The "White-Box" Reality: The paper assumes a white-box setting where the attacker knows the server's aggregation rule. Because many FL frameworks use open-source protocols, this remains a realistic threat model.

Results

  • Superior Performance: The RL-based attack consistently outperformed state-of-the-art baselines like Inner Product Manipulation (IPM) and Local Model Poisoning (LMP).
  • Total Collapse: Against the Krum defense on MNIST, the RL attack successfully drove the global model accuracy down to ~10% (essentially random guessing) almost immediately after the attack started.
  • Resilience to Subsampling: Even when only a small fraction of clients were malicious (as low as 8%), the RL attack remained effective, whereas baseline attacks failed to cause significant damage.
  • Defense Degradation: Against Clipping Median (a very strong defense), the RL attack still managed to reduce global accuracy to around 50%, while other attacks barely made a dent.

Limitations and Future Directions

While highly effective, the attack has limitations. Training an RL agent requires a simulator and significant compute, presenting a higher barrier to entry than simple heuristic attacks. Additionally, the current framework relies on knowing the server's aggregation rule (white-box dependency). The authors suggest meta-learning as a future direction to develop "black-box" attacks that can infer the defense mechanism.

Finally, RL training can be unstable; longer training doesn't always yield a better attack policy, indicating a need for better validation techniques for attackers. Ultimately, this paper serves as a wake-up call, highlighting the urgent need to develop adaptive defenses capable of detecting the long-term statistical shifts caused by RL-based adversaries.

Summary updated

Abstract

We propose a model-based reinforcement learning framework to derive untargeted poisoning attacks against federated learning (FL) systems. Our framework first approximates the distribution of the clients' aggregated data using model updates from the server. The learned distribution is then used to build a simulator of the FL environment, which is utilized to learn an adaptive attack policy through reinforcement learning. Our framework is capable of learning strong attacks automatically even when the server adopts a robust aggregation rule. We further derive an upper bound on the attacker's performance loss due to inaccurate distribution estimation. Experimental results on real-world datasets demonstrate that the proposed attack framework significantly outperforms state-of-the-art poisoning attacks. This indicates the importance of developing adaptive defenses for FL systems.
BibTeX
@inproceedings{li2022learning,
  author = {Henger Li and Xiaolin Sun and Zizhan Zheng},
  title = {{Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=uaumxx-WSE},
}
Figure