The Thieves on Sesame Street are Polyglots - Extracting Multilingual Models from Monolingual APIs

Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard SocherEMNLP 2020 · OpenReview

Abstract: Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

In the landscape of machine learning security, "model stealing" (or model extraction) has long been recognized as a threat to intellectual property. Typically, an adversary queries a proprietary API with numerous inputs, records the outputs, and trains a "copycat" model to mimic the victim. While previous research demonstrated that this could be achieved using random "gibberish" queries, new research from Salesforce reveals a significant escalation: an adversary can steal a monolingual model and transform it into a multilingual one.

This paper demonstrates that the task-specific logic of an English-only API can be extracted and "injected" into a pre-trained multilingual model. The resulting stolen model not only replicates the original API's performance in English but also gains the ability to perform the task in dozens of other languages—despite never seeing a single well-formed sentence in those languages during the extraction process.

Methodology: The Extraction Framework

The researchers focused on Natural Language Inference (NLI), a task requiring a model to determine if a "premise" sentence entails, contradicts, or is neutral toward a "hypothesis" sentence. The attack follows a two-phase process: Extraction and Inference.

The two-phase process of model extraction: capturing logic from an English victim and applying it to a multilingual extractor.
  1. The Victim: A high-performing RoBERTa-Large model trained on the English Multi-Genre NLI (MNLI) dataset.
  2. The Extractor: A pre-trained multilingual model (e.g., XLM-R) that possesses cross-lingual knowledge but lacks task-specific NLI training.
  3. The Attack: The attacker generates "gibberish" by sampling words from a general vocabulary (WikiText-103). They create pairs where the premise and hypothesis share most words but have random swaps to mimic NLI structures. These nonsense pairs are sent to the English Victim API to obtain "hard labels."
  4. Fine-tuning: The Extractor is fine-tuned on these (Gibberish, Label) pairs, effectively distilling the victim's decision boundaries into the extractor's multilingual embedding space.

Key Insights: Why Gibberish Works

The success of this method relies on the internal representation of pre-trained models. Analysis shows that these models produce very similar embeddings for real sentences and gibberish. Because the model perceives them similarly, it can learn the underlying logic of a task from nonsense just as effectively as from real data. This process "unlocks" the task across all languages the extractor was pre-trained on, generalizing the concept of NLI across its internal multilingual space.

However, vocabulary alignment is critical. If the gibberish vocabulary is from a different domain (e.g., using COVID-19 terminology to query a general-purpose model), extraction performance drops significantly. The "nonsense" must still reside within the general distribution the victim understands.

Experimental Results

The results confirm that an XLM-R model extracted using only English gibberish (0% real data) achieves remarkable zero-shot cross-lingual performance. As shown in the table below, the model maintains strong accuracy across 15 different languages without ever seeing a real sentence in them.

Performance metrics across 15 languages, showing the impact of adding small amounts of real data to the extraction process.
  • Zero-Shot Success: Using 0% real data, the model achieved 77.0% accuracy in English, 71.8% in French, and 68.3% in Russian.
  • The 5% Boost: Incorporating just 5% of real English data during or after extraction significantly improved results, bringing English accuracy to 86.2% and Spanish to 81.3%.
  • Post-hoc Refinement: Fine-tuning on real data after the gibberish extraction phase proved more effective than mixing real and gibberish data simultaneously.

Conclusion and Security Implications

This research highlights a major vulnerability in monolingual NLP APIs. A "polyglot" stolen model is significantly more valuable than the monolingual original, increasing the incentive for model theft. Furthermore, traditional defenses like watermarking may be ineffective, as the extractor model learns the underlying logic rather than mimicking specific phrasing. Future work must focus on building APIs capable of detecting extraction-style queries to protect high-value machine learning assets.

Summary updated

BibTeX
@inproceedings{keskar2020thieves,
  author = {Nitish Shirish Keskar and Bryan McCann and Caiming Xiong and Richard Socher},
  title = {{The Thieves on Sesame Street are Polyglots - Extracting Multilingual Models from Monolingual APIs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=XrBqRJDmJPh},
}
Figure