TL;DR: High-performing NLP models based on BERT can be "stolen" from black-box APIs for just a few hundred dollars. Surprisingly, an attacker doesn't even need real training data; they can successfully extract a victim model's functionality using nonsensical, random sequences of words as queries.
The Vulnerability of Black-Box APIs
Machine learning models are expensive to build, requiring massive datasets, specialized expertise, and significant compute time. To protect this intellectual property, companies often serve models behind "black-box" APIs. Users can submit inputs and receive predictions, but they cannot access the model's internal weights or architecture.
"Model extraction" is an attack where a malicious user queries an API repeatedly to train a local "clone" that mimics the victim's behavior. While this threat has been studied extensively in computer vision, text-based extraction was historically considered more difficult because the input space is discrete—you cannot easily "nudge" a word toward a decision boundary like you can with pixel values. However, this paper investigates whether the recent shift toward transfer learning—specifically fine-tuning large pre-trained models like BERT—has inadvertently made NLP APIs more vulnerable to extraction attacks.
How the Attack Works
The authors simulate an adversary who wants to extract a victim model (a fine-tuned BERT-large) without having access to the original training data. The attack relies on a two-step process: query generation and model extraction.
- Query Generation: The attacker generates synthetic queries using two main strategies:
- RANDOM: Nonsensical sequences of words sampled uniformly from a Wikipedia vocabulary.
- WIKI: Actual sentences or paragraphs taken from the WikiText-103 dataset.
- Task-Specific Heuristics: For complex tasks involving pairs of inputs (like Question Answering or Natural Language Inference), pure randomness is inefficient. The authors introduce simple heuristics, such as sampling words from a generated passage to form a "question," or randomly swapping words between a premise and a hypothesis.
- Extraction: The attacker sends these queries to the victim API, collects the predicted labels or probability distributions, and then fine-tunes their own pre-trained model (e.g., another BERT or XLNet) on this newly labeled synthetic dataset.
Key Insights: Why Gibberish Works
- Pre-training is an Achilles' Heel: The very feature that makes BERT powerful—its broad linguistic knowledge from pre-training—makes it easier to steal. Because both the victim and the attacker start with the same pre-trained "base," the attacker only needs a small amount of task-specific signal to align their model with the victim's logic.
- Grammar is Optional: Victim models are surprisingly "opinionated" about gibberish. Even when fed a string of random words, a fine-tuned BERT model will often produce a high-confidence prediction. These "rubbish" inputs provide enough signal for an attacker to map out the victim's decision boundaries.
- The Power of Agreement: The authors found that if multiple versions of a victim model (trained with different random seeds) agree on the answer to a nonsensical query, that query is highly effective for extraction.
- Architecture Mismatch: An attacker doesn't need to know the victim's exact architecture. Using a more powerful model (like XLNet) to extract a BERT model can sometimes result in a clone that performs nearly as well as the original.
Extraction Results and Costs
The study evaluates extraction across four diverse NLP benchmarks: SST2 (sentiment), MNLI (inference), SQuAD 1.1 (question answering), and BoolQ (boolean QA). The attacks proved remarkably successful and cost-effective.
- High Fidelity: On SQuAD, the attacker recovered 95% of the original model's accuracy (86.1 F1 vs. 90.6 F1) using only Wikipedia-based queries.
- Low Cost: Most extraction attacks cost less than $100 in API fees. The most complex attack (MNLI) cost roughly $388, which is a fraction of the cost required to develop such a model from scratch.
Defenses are Fragile
The authors tested two common defenses: Membership Classification (detecting "out-of-distribution" queries) and API Watermarking (intentionally providing wrong answers for certain queries to "mark" the model). They found that while these defenses can stop naive attackers, sophisticated adversaries can easily bypass them by slightly modifying their query strategy or training for more epochs.
Limitations and Future Work
While these attacks are highly effective, the extracted models are not "functionally equivalent" in every sense. They might achieve high accuracy on the original task but behave differently on other out-of-distribution data.
Furthermore, the defenses explored in this study were relatively simple. The authors suggest that future work should investigate more robust defenses, such as incorporating Differential Privacy during training or implementing advanced query monitoring that accounts for the "efficiency" of the information being extracted by the user.
Summary updated