Thieves on Sesame Street! Model Extraction of BERT-based APIs

Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer

Abstract

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against naive adversaries, are ineffective against more sophisticated ones.

Introduction

Machine learning models represent significant intellectual property, requiring extensive datasets and computational resources to develop. To protect these assets, companies often deploy their models behind black-box APIs, allowing users to receive outputs for specific inputs without exposing internal weights. However, "model extraction" (or model stealing) poses a significant threat where an adversary queries the API to observe input-output relationships and trains a local "copycat" model. This research investigates how the widespread adoption of large, pretrained models like BERT has influenced the feasibility and cost of such extraction attacks in the NLP domain.

Methodology

The proposed attack strategy leverages the shared knowledge inherent in pretrained language models. The adversary assumes the victim has fine-tuned a BERT model for a specific task. The extraction process follows a two-step approach:

  1. Query Generation: The attacker generates queries using either nonsensical word sequences sampled from Wikipedia (RANDOM) or actual Wikipedia sentences unrelated to the target task (WIKI). Simple task-specific heuristics are applied to these queries to mimic the expected input structure (e.g., adding "what" and "?" for question-answering tasks).
  2. Model Fine-tuning: These queries are sent to the victim API. The resulting outputs are used as labels to fine-tune the attacker's own local BERT model.
Overview of the model extraction attack process, showing query generation and local model fine-tuning.

The study evaluates this attack across four benchmarks: SST2 (sentiment), MNLI (inference), SQuAD 1.1 (extractive QA), and BoolQ (boolean QA).

Results and Analysis

The attacks proved remarkably effective across all tested tasks. As shown in the results table, the extracted models recovered a significant portion of the victim models' performance. For instance, on the SST2 task, the victim's 93.1% accuracy was nearly matched by a RANDOM-query-based clone at 90.1%. On SQuAD, the victim's 90.6 F1 score was recovered to 86.1 F1 using Wikipedia-based queries.

Performance and cost metrics for model extraction across various NLP tasks.

Key findings include:

  • Low Barrier to Entry: Successful extraction can be achieved for as little as $50 to $500 using commercial API rates.
  • The Role of Pretraining: Because both the victim and attacker start with a "language-aware" base (BERT), the attacker only needs to extract the thin task-specific layer, drastically reducing the required number of queries.
  • Effectiveness of Nonsense: Queries do not need to be semantically meaningful; random "word salads" are sufficient to probe decision boundaries.
  • Defense Vulnerability: Existing defenses like membership classification and API watermarking are easily bypassed by sophisticated adversaries who can ignore watermarks or generate queries that evade detection.

Conclusion

The shift toward transfer learning in NLP has inadvertently lowered the sample complexity for model extraction attacks. While pretraining provides immense performance benefits, it also makes proprietary models more vulnerable to theft. Future research must focus on developing proactive defenses that address the fundamental vulnerabilities exposed by these "rubbish" input attacks.

Summary updated

Abstract

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.
BibTeX
@inproceedings{krishna2020thieves,
  author = {Kalpesh Krishna and Gaurav Singh Tomar and Ankur P. Parikh and Nicolas Papernot and Mohit Iyyer},
  title = {{Thieves on Sesame Street! Model Extraction of BERT-based APIs}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=HZamF_OUg9},
}
Figure