Fabricated Flips: Poisoning Federated Learning without Data

Jiyue Huang, Zilong Zhao, Lydia Y. Chen, Stefanie Roos

Abstract

Attacks on Federated Learning (FL) can severely reduce the quality of the generated models and limit the usefulness of this emerging learning paradigm that enables on-premise decentralized learning. However, existing untargeted attacks are not practical for many scenarios as they assume that i) the attacker knows every update of benign clients, or ii) the attacker has a large dataset to locally train updates imitating benign parties. In this paper, we propose a data-free untargeted attack (DFA) that synthesizes malicious data to craft adversarial models without eavesdropping on the transmission of benign clients at all or requiring a large quantity of task-specific training data. We design two variants of DFA, namely DFA-R and DFA-G, which differ in how they trade off stealthiness and effectiveness. Specifically, DFA-R iteratively optimizes a malicious data layer to minimize the prediction confidence of all outputs of the global model, whereas DFA-G interactively trains a malicious data generator network by steering the output of the global model toward a particular class. Experimental results on Fashion-MNIST, Cifar-10, and SVHN show that DFA, despite requiring fewer assumptions than existing attacks, achieves similar or even higher attack success rate than state-of-the-art untargeted attacks against various state-of-the-art defense mechanisms. Concretely, they can evade all considered defense mechanisms in at least 50% of the cases for CIFAR-10 and often reduce the accuracy by more than a factor of 2. Consequently, we design REFD, a defense specifically crafted to protect against data-free attacks. REFD leverages a reference dataset to detect updates that are biased or have a low confidence. It greatly improves upon existing defenses by filtering out the malicious updates and achieves high global model accuracy.

Introduction

Federated Learning (FL) is a promising paradigm for privacy-preserving machine learning, allowing multiple clients to collaboratively train a model without sharing their raw data. However, the decentralized nature of FL introduces significant security vulnerabilities. Malicious clients can participate in the training process to poison the global model, either by degrading its overall performance (untargeted attacks) or by introducing specific backdoors (targeted attacks).

Most existing untargeted attacks rely on heavy assumptions that limit their real-world applicability. "Informed" attackers are often assumed to have access to the updates of benign clients or possess large proxy datasets. In many sensitive domains, such as healthcare, these assumptions are unrealistic. This research introduces the Data-Free Attack (DFA), demonstrating that an adversary can launch devastating attacks without any real data or eavesdropping capabilities.

The Data-Free Attack (DFA) Framework

The core innovation of DFA is the use of synthetic data to craft adversarial model updates. Instead of relying on real images, the attacker optimizes synthetic inputs to exploit the decision boundaries of the current global model.

Overview of the Data-Free Attack (DFA) framework, showing how an adversary generates synthetic images to craft malicious updates without real data.

The researchers developed two distinct strategies for generating these "fabricated flips":

DFA-R: The Confuser

DFA-R focuses on maximizing the uncertainty of the global model. It utilizes a "filter layer" to transform random noise into synthetic images. Through iterative optimization, the filter is tuned so that the global model produces nearly uniform probability distributions across all classes for these synthetic inputs. When the global model is updated with these "ambiguous" samples, its discriminative power is severely compromised.

DFA-G: The Generator

DFA-G employs a Generator Network (typically a Transpose CNN) to create images that the global model classifies into a specific target class with high confidence. The attacker then assigns these images incorrect labels—a "fabricated flip"—and uses them to train a malicious update. This method actively steers the global model's decision boundaries toward incorrect classifications.

Comparison with Existing Attacks

DFA stands out because it operates under significantly fewer constraints than prior state-of-the-art (SOTA) attacks. While methods like LIE or Fang require knowledge of benign updates or raw data, DFA is entirely self-contained.

Comparison of DFA against state-of-the-art untargeted attacks, highlighting its minimal requirements and optimization-based approach.

As shown in the comparison, DFA is the only attack that handles data heterogeneity while requiring neither benign updates nor raw data. Furthermore, it shifts the attack type from simple statistics to a more sophisticated optimization-based approach.

Stealth and Defense

To evade detection, DFA incorporates a distance-based regularization term. This ensures that the malicious weights remain close to the previous global model, allowing them to bypass standard outlier-detection defenses like mKrum or Bulyan.

To counter this, the authors propose REFD (Reference-based Defense). Unlike distance-based defenses, REFD evaluates the behavior of updates using a tiny, clean reference dataset on the server. It calculates a "D-score" based on prediction confidence and class balance. Updates that result in low-confidence or highly biased predictions on the reference set are flagged and filtered out.

Key Results and Conclusion

Experiments on Fashion-MNIST, CIFAR-10, and SVHN benchmarks reveal that DFA can reduce model accuracy by more than 50%, even in the presence of robust defenses. On CIFAR-10, for instance, accuracy often plummeted from 50% to below 25%.

The proposed REFD defense proved highly effective, restoring accuracy to near-baseline levels by successfully identifying the subtle behavioral anomalies of data-free poisoned updates. This work highlights a critical new vector in FL security: the ability to sabotage learning systems using purely synthetic, mathematically optimized noise.

Summary updated

Abstract

Attacks on Federated Learning (FL) can severely reduce the quality of the generated models and limit the usefulness of this emerging learning paradigm that enables on-premise decentralized learning. However, existing untargeted attacks are not practical for many scenarios as they assume that i) the attacker knows every update of benign clients, or ii) the attacker has a large dataset to locally train updates imitating benign parties. In this paper, we propose a data-free untargeted attack (DFA) that synthesizes malicious data to craft adversarial models without eavesdropping on the transmission of benign clients at all or requiring a large quantity of task-specific training data. We design two variants of DFA, namely DFA-R and DFA-G, which differ in how they trade off stealthiness and effectiveness. Specifically, DFA-R iteratively optimizes a malicious data layer to minimize the prediction confidence of all outputs of the global model, whereas DFA-G interactively trains a malicious data generator network by steering the output of the global model toward a particular class. Experimental results on Fashion-MNIST, Cifar-10, and SVHN show that DFA, despite requiring fewer assumptions than existing attacks, achieves similar or even higher attack success rate than state-of-the-art untargeted attacks against various state-of-the-art defense mechanisms. Concretely, they can evade all considered defense mechanisms in at least 50% of the cases for CIFAR-10 and often reduce the accuracy by more than a factor of 2. Consequently, we design REFD, a defense specifically crafted to protect against data-free attacks. REFD leverages a reference dataset to detect updates that are biased or have a low confidence. It greatly improves upon existing defenses by filtering out the malicious updates and achieves high global model accuracy.
BibTeX
@inproceedings{huang2023fabricated,
  author = {Jiyue Huang and Zilong Zhao and Lydia Y. Chen and Stefanie Roos},
  title = {{Fabricated Flips: Poisoning Federated Learning without Data}},
  booktitle = {DSN},
  year = {2023},
  url = {https://openreview.net/forum?id=CsLbLcUHpy},
}
Figure