SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-supervised Learning

Peizhuo Lv, Pan Li, Shenchen Zhu, Shengzhi Zhang, Kai Chen, Ruigang Liang, Chang Yue, Xiang Fan, Yuling Cai, Hualong Ma, Ying–Jun Angela Zhang, Guozhu Meng

Abstract

Recent years have witnessed tremendous success in Self-Supervised Learning (SSL), which has been widely utilized to facilitate various downstream tasks in Computer Vision (CV) and Natural Language Processing (NLP) domains. However, attackers may steal such SSL models and commercialize them for profit, making it crucial to verify the ownership of the SSL models. Most existing ownership protection solutions (e.g., backdoor-based watermarks) are designed for supervised learning models and cannot be used directly since they require that the models' downstream tasks and target labels be known and available during watermark embedding, which is not always possible in the domain of SSL. To address such a problem, especially when downstream tasks are diverse and unknown during watermark embedding, we propose a novel black-box watermarking solution, named SSL-WM, for verifying the ownership of SSL models. SSL-WM maps watermarked inputs of the protected encoders into an invariant representation space, which causes any downstream classifier to produce expected behavior, thus allowing the detection of embedded watermarks. We evaluate SSL-WM on numerous tasks, such as CV and NLP, using different SSL models both contrastive-based and generative-based. Experimental results demonstrate that SSL-WM can effectively verify the ownership of stolen SSL models in various downstream tasks. Furthermore, SSL-WM is robust against model fine-tuning, pruning, and input preprocessing attacks. Lastly, SSL-WM can also evade detection from evaluated watermark detection approaches, demonstrating its promising application in protecting the ownership of SSL models.

Introduction

Self-supervised learning (SSL) has revolutionized artificial intelligence by enabling models like GPT-3 or CLIP to learn from massive, unlabeled datasets. However, these models represent significant intellectual property; training a model like GPT-3 can cost upwards of $12 million in compute alone. This high value makes SSL encoders prime targets for "model stealing," where an attacker illicitly acquires the encoder to repurpose it for commercial gain.

Protecting these assets is challenging. Traditional watermarking techniques, such as backdoor-based triggers, typically fail in the SSL context for two primary reasons. First, downstream tasks are often unknown during the pre-training phase. Unlike supervised models, an SSL encoder is a general-purpose feature extractor; the owner cannot predict if a thief will use it for sentiment analysis, medical imaging, or autonomous driving. Second, owners usually only have "black-box" access to a suspect model, meaning they can observe final classification outputs but cannot inspect the internal weights or intermediate feature vectors produced by the encoder.

Methodology

To overcome these hurdles, the proposed SSL-WM framework shifts the focus from specific target labels to invariant representations. The framework operates in two distinct phases: embedding and verification.

Watermark Embedding

During the pre-training phase, the model owner incorporates a specific watermark loss ($L_{wm}$) alongside the standard training objective. This loss is designed to force the encoder to map any input containing a specific "trigger" pattern into a highly concentrated cluster within the latent (feature) space. By using multi-domain shadow datasets (e.g., combining traffic signs, movie reviews, and general imagery), the owner ensures the watermark remains effective even if the encoder is applied to a dataset previously unseen by the owner.

Verification via Behavioral Consistency

Because all watermarked inputs are forced into the same feature cluster, any downstream classifier built upon the stolen encoder will likely assign these samples to the same class, regardless of what that class represents. This behavior results in very low Shannon Entropy across the outputs for watermarked samples. To verify ownership, the owner queries the suspect model with both clean and watermarked samples and applies Median Absolute Deviation (MAD) to determine if the entropy of the watermarked samples is a statistical outlier. If the "outlier index" exceeds a predefined threshold (typically set at 3), ownership is confirmed.

The SSL-WM framework overview

Experimental Evaluation

SSL-WM was evaluated across a diverse set of benchmarks in both Computer Vision (CV) and Natural Language Processing (NLP). Tested architectures included contrastive-based models like SimCLR, MoCoV2, BYOL, and CLIP, as well as generative-based models like BiGAN and BERT. Downstream tasks ranged from image classification (CIFAR-10, STL-10, GTSRB) to sentiment analysis (IMDB) and natural language inference (SNLI).

Key Results

Experimental results demonstrate that SSL-WM is both highly reliable and stealthy. The average outlier index for watermarked models was recorded at 33.50, significantly higher than the threshold of 3, while clean models never exceeded an index of 1.22. This clear separation ensures zero false positives in ownership verification.

Experimental results across various SSL models and downstream tasks

Furthermore, the utility impact is negligible, with only a 0.41% average drop in primary task accuracy. The watermark also proved robust against common model "laundering" attempts, including aggressive fine-tuning and parameter pruning (remaining detectable even after 90% pruning). Finally, SSL-WM successfully evaded five state-of-the-art backdoor detection frameworks, including Neural Cleanse and ABS, due to its influence being distributed across numerous neurons rather than a single high-activation trigger.

Conclusion

SSL-WM represents the first generic, black-box watermarking solution designed specifically for the unique constraints of self-supervised learning. By leveraging representation invariance and entropy-based outlier detection, it provides a robust and stealthy mechanism for protecting high-value AI assets across unknown downstream applications. Future work will continue to explore defenses against adaptive attackers who may attempt to mask these statistical signatures through entropy-smoothing techniques.

Summary updated

Abstract

Recent years have witnessed tremendous success in Self-Supervised Learning (SSL), which has been widely utilized to facilitate various downstream tasks in Computer Vision (CV) and Natural Language Processing (NLP) domains. However, attackers may steal such SSL models and commercialize them for profit, making it crucial to verify the ownership of the SSL models. Most existing ownership protection solutions (e.g., backdoor-based watermarks) are designed for supervised learning models and cannot be used directly since they require that the models' downstream tasks and target labels be known and available during watermark embedding, which is not always possible in the domain of SSL. To address such a problem, especially when downstream tasks are diverse and unknown during watermark embedding, we propose a novel black-box watermarking solution, named SSL-WM, for verifying the ownership of SSL models. SSL-WM maps watermarked inputs of the protected encoders into an invariant representation space, which causes any downstream classifier to produce expected behavior, thus allowing the detection of embedded watermarks. We evaluate SSL-WM on numerous tasks, such as CV and NLP, using different SSL models both contrastive-based and generative-based. Experimental results demonstrate that SSL-WM can effectively verify the ownership of stolen SSL models in various downstream tasks. Furthermore, SSL-WM is robust against model fine-tuning, pruning, and input preprocessing attacks. Lastly, SSL-WM can also evade detection from evaluated watermark detection approaches, demonstrating its promising application in protecting the ownership of SSL models.
BibTeX
@misc{lv2022ssl,
  author = {Peizhuo Lv and Pan Li and Shenchen Zhu and Shengzhi Zhang and Kai Chen and Ruigang Liang and Chang Yue and Xiang Fan and Yuling Cai and Hualong Ma and Ying–Jun Angela Zhang and Guozhu Meng},
  title = {{SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-supervised Learning}},
  year = {2022},
  eprint = {2209.03563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.03563},
}
Figure