Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation

Hong-Min Chu, Jonas Geiping, Liam H Fowl, Micah Goldblum, Tom Goldstein

This paper, presented at ICLR 2023, introduces a formidable new privacy threat to Federated Learning (FL) systems. It demonstrates how a malicious server can bypass standard privacy protections to extract specific, sensitive text from users—even when their data is buried under the "noise" of thousands of other participants.

The Threat at a Glance

Researchers have developed the first "aggregation-agnostic" attack on Federated Learning that extracts targeted sensitive phrases (like credit card numbers) by maliciously reprogramming a Transformer model's parameters. Unlike previous attacks that fail as more users are added, this "panning" method remains highly effective regardless of how many users' gradients are averaged together.

What Problem Does This Paper Address?

Federated Learning is often touted for its privacy benefits: instead of sending raw data to a central server, users train a model locally and only share "gradients" (mathematical updates). To further protect privacy, servers typically use Secure Aggregation, which averages the updates from hundreds or thousands of users so that no single user’s contribution can be identified.

Existing privacy attacks (known as gradient inversion) usually try to reconstruct all the data in a batch. These attacks work well for small groups but break down completely in "industrial-scale" settings where the aggregated gradient becomes a blurry mess of information. The authors identify a critical gap: an attacker doesn't need to see everything; they just need to find the "gold"—the rare, sensitive snippets like "my password is..." or "SSN: ...".

Methods: Turning the Model into a Filter

The authors propose a "malicious server" threat model. Before the training round begins, the server sends a specially crafted, "poisoned" version of the Transformer model to users. This model is designed to act as a filter through three key steps:

Visual representation of the tagging and filtering process used to isolate targeted sequences like credit card numbers.
  1. Keyword Tagging (MHSA): The attacker modifies the Multi-Head Self-Attention (MHSA) layers. When the model encounters a "trigger" keyword (e.g., "credit"), it forces every other token in that sentence to attend to it. This "imprints" a specific mathematical signature (a tag) onto the embeddings of the entire sequence.
  2. Gradient Filtering (Linear Layers): The attacker reprograms the feed-forward linear layers to act as a "thermometer." Using a clever trick with ReLU activation functions and biased weights, the model is told to only generate significant gradients for tokens that carry the "imprint" from step one. All other "uninteresting" text is effectively silenced.
  3. Post-Processing: Because the resulting gradient update is sparse and contains only the "tagged" information, the attacker can use bipartite matching algorithms to reconstruct the original tokens and their positions with high fidelity.

Key Insights and Ideas

  • Scale Invariance: The most critical insight is that by narrowing the attack's focus, the attacker makes the attack stronger. Because sensitive phrases are rare, the aggregated gradient for those specific phrases doesn't get "blurry"—it stays sharp while the rest of the data is filtered out.
  • The Model as a Search Engine: Instead of trying to reverse-engineer a standard model, the attacker turns the model itself into a search-and-filter tool that does the heavy lifting on the user's own device.
  • Bimodal Distribution: The attack creates a clear separation in the measurement distribution. Targeted embeddings produce massive activations, while irrelevant ones produce near-zero activations, making the "gold" incredibly easy to spot.

Key Results

The researchers evaluated their attack using standard NLP datasets (Wikitext) and tested it on variants of GPT-2 and BERT to show the attack works on both causal and bidirectional architectures. They compared their "Panning" attack against Decepticons, a state-of-the-art previous attack.

Comparison of extraction performance (ROUGE-L) across increasing batch sizes, demonstrating the attack's resilience to scale.
  • Superior Scaling: While Decepticons' accuracy plummeted once the batch size (number of sequences) exceeded 128, the Panning attack maintained near-perfect reconstruction accuracy even at a batch size of 2048.
  • High Fidelity: On targeted sequences, the attack achieved near-perfect BLEU and ROUGE scores, meaning the extracted text was almost identical to the original private data.
  • Multi-Targeting: The attack successfully extracted sequences triggered by up to 11 different keywords simultaneously, covering a wide net of potentially sensitive information.

Limitations and Future Work

  • Malicious Server Requirement: This is an "active" attack. It requires the server to be able to modify model parameters. It does not apply to "honest-but-curious" servers who only observe standard training.
  • Differential Privacy (DP): The authors note that strong Differential Privacy (adding noise and clipping gradients) can mitigate the attack. However, they point out that DP often comes at the cost of model performance, creating a difficult trade-off for practitioners.
  • Detection: While the malicious parameters are "conspicuous" (containing very large values), current FL frameworks lack automated tools for users to inspect and vet the models they receive from a server. Future work may involve developing "parameter firewalls" to detect these malicious signatures before training begins.

Summary updated

Abstract

As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex. Attacks on language models have progressed from recovering single sentences in simple classification tasks to recovering larger parts of user data. Current attacks against federated language models are sequence-agnostic and aim to extract as much data as possible from an FL update - often at the expense of fidelity for any particular sequence. Because of this, current attacks fail to extract any meaningful data under large-scale aggregation. In realistic settings, an attacker cares most about a small portion of user data that contains sensitive personal information, for example sequences containing the phrase "my credit card number is ...". In this work, we propose the first attack on FL that achieves targeted extraction of sequences that contain privacy-critical phrases, whereby we employ maliciously modified parameters to allow the transformer itself to filter relevant sequences from aggregated user data and encode them in the gradient update. Our attack can effectively extract sequences of interest even against extremely large-scale aggregation.
BibTeX
@inproceedings{chu2023panning,
  author = {Hong-Min Chu and Jonas Geiping and Liam H Fowl and Micah Goldblum and Tom Goldstein},
  title = {{Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=A9WQaxYsfx},
}
Figure