Stealing and Defending Transformer-based Encoders

Adam Dziedzic, Franziska Boenisch, Mingjian Jiang, Haonan Duan, Nicolas PapernotICLR 2023 · OpenReview

Abstract

Self-supervised learning (SSL) has become the predominant approach to training on large amounts of unlabeled data. New real-world APIs offer services to generate high-dimensional representations for given inputs based on SSL encoders with transformer architectures. Recent efforts highlight that it is possible to steal high-quality SSL encoders trained on convolutional neural networks. In this work, we are the first to extend this line of work to stealing and defending transformer-based encoders in both language and vision domains. We show that it is possible to steal transformer-based sentence embedding models solely using their returned representations and with 40x fewer queries than the number of victim's training data points. We also decrease the number of required stealing queries for the vision encoders by leveraging semi-supervised learning. Finally, to defend vision transformers against stealing attacks, we propose a defense technique that combines watermarking with dataset inference. Our method creates a unique encoder signature based on a private data subset that acts as a secret seed during training. By applying dataset inference on the seed, we can then successfully identify stolen transformers.
BibTeX
@inproceedings{dziedzic2023stealing,
  author = {Adam Dziedzic and Franziska Boenisch and Mingjian Jiang and Haonan Duan and Nicolas Papernot},
  title = {{Stealing and Defending Transformer-based Encoders}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=LoJ6oXzc_P3},
}

Loading related content...

Figure