Fabricated Flips: Poisoning Federated Learning without Data

Jiyue Huang, Zilong Zhao, Lydia Y. Chen, Stefanie Roos

TL;DR

Researchers have developed a novel "Data-Free Attack" (DFA) that can cripple Federated Learning (FL) systems without requiring any real training data or the ability to eavesdrop on other users. By synthesizing malicious images designed to confuse or mislead the global model, the authors demonstrate that attackers can bypass state-of-the-art defenses and propose a new defense, REFD, that uses a small reference dataset to identify these "fabricated" updates.

What Problem Does This Paper Address?

Federated Learning (FL) is designed to train machine learning models across decentralized devices (like smartphones or medical sensors) without ever sharing raw data. While this protects privacy, it creates a massive security loophole: the central server cannot verify if a client’s update is based on real, high-quality data or malicious "poison."

Existing "untargeted" attacks—which aim to degrade the overall accuracy of the model—usually rely on two unrealistic assumptions:

  1. Eavesdropping: The attacker can see the updates sent by benign (honest) clients.
  2. Data Wealth: The attacker has a large, task-specific dataset to train their own malicious updates.

In many real-world scenarios (like classifying rare diseases), an attacker has neither. This paper asks: Can an attacker destroy a global model using zero real data and zero knowledge of other clients?

Methods

The authors introduce the Data-Free Attack (DFA), which uses the feedback from the global model itself to "hallucinate" malicious training data. They propose two flavors:

  • DFA-R (Filter-based): This variant adds a "filter layer" to a dummy image. It iteratively optimizes this filter so that the global model becomes as "unsure" as possible about the resulting image (i.e., it assigns an equal probability to every possible class). Training on this "ambiguous" data effectively "unlearns" the model's certainty.
  • DFA-G (Generator-based): This uses a Generative Adversarial Network (GAN) approach. A generator network learns to create noisy images that the global model thinks belong to a specific class. The attacker then takes these images, pairs them with incorrect labels, and trains a local model to submit to the server.

To stay "stealthy" and avoid being flagged as an outlier, both methods use Distance-based Regularization. This ensures the malicious update doesn't drift too far from the previous global model, mimicking the natural step-sizes of honest clients.

On the defensive side, the authors propose REFD. Since the server can't see client data, REFD uses a tiny, clean "reference dataset" at the server. It calculates a D-score for every incoming update based on:

  1. Balance: Does the update make the model biased toward one specific class?
  2. Confidence: Does the update make the model's predictions generally weaker?

Datasets

The attack and defense were evaluated on three standard computer vision benchmarks:

  • Fashion-MNIST: Grayscale images of clothing.
  • CIFAR-10: Color images of common objects (planes, dogs, etc.).
  • SVHN: Street View House Numbers.

The researchers simulated "Data Heterogeneity" (where different clients have different types of data) using a Dirichlet distribution ($\beta=0.5$), which is a standard way to model realistic, messy FL environments.

Key Insights and Ideas

  • The Model is the Data: The most surprising insight is that the global model contains enough "knowledge" for an attacker to reconstruct the very data needed to destroy it. You don't need to steal data if you can probe the model to see what it finds confusing.
  • Synthetic > Real: In some tests, synthetic malicious data was more effective than using real data with flipped labels. This is because the synthetic data is mathematically optimized to exploit the specific weaknesses of the current global model.
  • Stealth via Proximity: Traditional defenses look for "wild" updates. By adding a regularization term that keeps malicious updates within a "normal" distance from the starting point, the attackers proved that "statistical outliers" are not the only way to poison a system.
  • The D-Score Synergy: The REFD defense works because it recognizes that poisoning usually manifests in one of two ways: either the model becomes a "fanatic" (biased toward one class) or a "skeptic" (low confidence in everything). By combining these into one score, they caught attacks that individual metrics missed.

Key Results

  • Attack Potency: DFA-R and DFA-G successfully reduced global model accuracy by more than 50% in many settings, even when "robust" aggregation defenses like mKrum or Bulyan were active.
  • Defense Efficacy: The proposed REFD defense was a game-changer. On Fashion-MNIST under a DFA-R attack, accuracy dropped to ~40% with standard defenses; REFD brought it back up to 70-75%, nearly the level of a clean, unattacked model.
  • Efficiency: The attacks are computationally "cheap," requiring only a few local epochs to generate effective malicious updates.

Limitations and Future Work

  • Reference Data Requirement: REFD assumes the server has access to a small, clean reference dataset. In extremely sensitive domains (like private financial data), even a small "public" reference set might be hard to obtain.
  • Hyperparameter Sensitivity: The D-score relies on a parameter ($\alpha$) to balance bias vs. confidence. While the authors used $\alpha=1$, they acknowledge that an adaptive $\alpha$ that learns over time could be more robust.
  • Beyond Images: The current work focuses on image classification. The authors suggest that future work should explore how these data-free "fabricated flips" might work in Natural Language Processing (NLP) or on tabular data.

Summary updated

Abstract

Attacks on Federated Learning (FL) can severely reduce the quality of the generated models and limit the usefulness of this emerging learning paradigm that enables on-premise decentralized learning. However, existing untargeted attacks are not practical for many scenarios as they assume that i) the attacker knows every update of benign clients, or ii) the attacker has a large dataset to locally train updates imitating benign parties. In this paper, we propose a data-free untargeted attack (DFA) that synthesizes malicious data to craft adversarial models without eavesdropping on the transmission of benign clients at all or requiring a large quantity of task-specific training data. We design two variants of DFA, namely DFA-R and DFA-G, which differ in how they trade off stealthiness and effectiveness. Specifically, DFA-R iteratively optimizes a malicious data layer to minimize the prediction confidence of all outputs of the global model, whereas DFA-G interactively trains a malicious data generator network by steering the output of the global model toward a particular class. Experimental results on Fashion-MNIST, Cifar-10, and SVHN show that DFA, despite requiring fewer assumptions than existing attacks, achieves similar or even higher attack success rate than state-of-the-art untargeted attacks against various state-of-the-art defense mechanisms. Concretely, they can evade all considered defense mechanisms in at least 50% of the cases for CIFAR-10 and often reduce the accuracy by more than a factor of 2. Consequently, we design REFD, a defense specifically crafted to protect against data-free attacks. REFD leverages a reference dataset to detect updates that are biased or have a low confidence. It greatly improves upon existing defenses by filtering out the malicious updates and achieves high global model accuracy
BibTeX
@inproceedings{huang2023fabricated,
  author = {Jiyue Huang and Zilong Zhao and Lydia Y. Chen and Stefanie Roos},
  title = {{Fabricated Flips: Poisoning Federated Learning without Data}},
  booktitle = {DSN},
  year = {2023},
  url = {https://openreview.net/forum?id=CsLbLcUHpy},
}
Figure