Abstract
We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against naive adversaries, are ineffective against more sophisticated ones.
Introduction
Machine learning models represent significant intellectual property, requiring extensive datasets and computational resources to develop. To protect these assets, companies often deploy their models behind black-box APIs, allowing users to receive outputs for specific inputs without exposing internal weights. However, "model extraction" (or model stealing) poses a significant threat where an adversary queries the API to observe input-output relationships and trains a local "copycat" model. This research investigates how the widespread adoption of large, pretrained models like BERT has influenced the feasibility and cost of such extraction attacks in the NLP domain.
Methodology
The proposed attack strategy leverages the shared knowledge inherent in pretrained language models. The adversary assumes the victim has fine-tuned a BERT model for a specific task. The extraction process follows a two-step approach:
- Query Generation: The attacker generates queries using either nonsensical word sequences sampled from Wikipedia (RANDOM) or actual Wikipedia sentences unrelated to the target task (WIKI). Simple task-specific heuristics are applied to these queries to mimic the expected input structure (e.g., adding "what" and "?" for question-answering tasks).
- Model Fine-tuning: These queries are sent to the victim API. The resulting outputs are used as labels to fine-tune the attacker's own local BERT model.
The study evaluates this attack across four benchmarks: SST2 (sentiment), MNLI (inference), SQuAD 1.1 (extractive QA), and BoolQ (boolean QA).
Results and Analysis
The attacks proved remarkably effective across all tested tasks. As shown in the results table, the extracted models recovered a significant portion of the victim models' performance. For instance, on the SST2 task, the victim's 93.1% accuracy was nearly matched by a RANDOM-query-based clone at 90.1%. On SQuAD, the victim's 90.6 F1 score was recovered to 86.1 F1 using Wikipedia-based queries.
Key findings include:
- Low Barrier to Entry: Successful extraction can be achieved for as little as $50 to $500 using commercial API rates.
- The Role of Pretraining: Because both the victim and attacker start with a "language-aware" base (BERT), the attacker only needs to extract the thin task-specific layer, drastically reducing the required number of queries.
- Effectiveness of Nonsense: Queries do not need to be semantically meaningful; random "word salads" are sufficient to probe decision boundaries.
- Defense Vulnerability: Existing defenses like membership classification and API watermarking are easily bypassed by sophisticated adversaries who can ignore watermarks or generate queries that evade detection.
Conclusion
The shift toward transfer learning in NLP has inadvertently lowered the sample complexity for model extraction attacks. While pretraining provides immense performance benefits, it also makes proprietary models more vulnerable to theft. Future research must focus on developing proactive defenses that address the fundamental vulnerabilities exposed by these "rubbish" input attacks.
Summary updated