Black-box Ownership Verification for Dataset Protection via Backdoor Watermarking

Yiming Li, Mingyan Zhu, Xue Yang, Yong Jiang, Shu-Tao Xia

Abstract

The rapid advancement of deep learning relies heavily on high-quality, large-scale datasets. However, protecting the intellectual property (IP) of these datasets remains a significant challenge, as traditional watermarking techniques often fail when only the final trained model is accessible. This paper introduces DVBW (Dataset Verification via Backdoor Watermarking), a novel framework for black-box ownership verification. By embedding "poison-only" backdoor triggers into a small fraction of the training data, DVBW enables data owners to prove a model was trained on their dataset through statistical hypothesis testing of model predictions. Our evaluation across image, text, and graph modalities shows that DVBW achieves high verification accuracy with negligible impact on primary task performance and remains robust against common model-level defenses.

Introduction

Data is the cornerstone of modern artificial intelligence. While datasets are frequently released under restrictive licenses, enforcing these terms is difficult once the data has been used to train a proprietary model. Existing protection methods, such as encryption or differential privacy, often degrade data utility or are inapplicable when the "thief" only provides black-box access to the resulting model. The authors address this by repurposing backdoor attacks—typically viewed as a security threat—into a proactive defense mechanism for dataset IP protection.

Methodology: The DVBW Framework

The DVBW pipeline consists of two primary phases: dataset watermarking and ownership verification.

The DVBW pipeline: Step 1 involves embedding triggers into a subset of the dataset, and Step 2 uses statistical tests on a suspicious model's output to verify ownership.

1. Dataset Watermarking

In the first phase, a defender selects a small subset of the benign dataset ($D_s$) and applies a "poison-only" backdoor attack. This involves adding a specific trigger (e.g., a pixel pattern or a keyword) to the samples and changing their labels to a pre-defined "target label." When a third party trains a model on this released victim dataset, the model inadvertently learns to associate the trigger with the target label, effectively embedding a hidden watermark.

2. Dataset Verification

To verify a suspicious model, the defender queries it with watermarked samples and analyzes the outputs:

  • Probability-Available Verification: When confidence scores are accessible, a pair-wise T-test compares the model's confidence in the target label for triggered vs. benign samples.
  • Label-Only Verification: In more restrictive settings where only class labels are provided, a Wilcoxon-test is employed to detect the presence of the backdoor statistically.

Experimental Evaluation

The authors evaluated DVBW across diverse data modalities to ensure its generalizability, including image recognition (CIFAR-10, ImageNet), Natural Language Processing (IMDB, DBpedia), and graph recognition (COLLAB, REDDIT).

Examples of benign and watermarked images using BadNets and Blended triggers on CIFAR-10 and ImageNet datasets.

As shown in Figure 3, the triggers used in image datasets (like BadNets or Blended patterns) are designed to be subtle, ensuring the dataset's utility for legitimate users remains high while remaining detectable by the owner.

Key Results and Insights

  • High Verification Success: The Watermark Success Rate (WSR) consistently exceeded 94%, reaching over 99% in many configurations. This indicates that models trained on watermarked data reliably exhibit the backdoor behavior.
  • Minimal Utility Loss: The impact on Benign Accuracy (BA) was minimal, typically dropping by less than 1%. This confirms that DVBW does not significantly hinder the primary training task.
  • Robustness: The embedded watermarks proved resilient to common post-training modifications, including model pruning and fine-tuning of the final layers.
  • Statistical Significance: In "steal" scenarios, the p-values obtained during verification were extremely low (e.g., $10^{-100}$), providing overwhelming evidence of unauthorized data use.

Conclusion and Future Work

DVBW successfully transforms backdoor attacks into a robust tool for dataset IP protection. By working in a black-box setting across multiple modalities, it offers a practical solution for data owners to monitor and enforce licensing agreements. Future research will focus on increasing the stealthiness of watermarks against adaptive defenses like Anti-Backdoor Learning (ABL) and optimizing the poisoning rate to balance verification strength with model utility.

Summary updated

Abstract

Deep learning, especially deep neural networks (DNNs), has been widely and successfully adopted in many critical applications for its high effectiveness and efficiency. The rapid development of DNNs has benefited from the existence of some high-quality datasets ($e.g.$, ImageNet), which allow researchers and developers to easily verify the performance of their methods. Currently, almost all existing released datasets require that they can only be adopted for academic or educational purposes rather than commercial purposes without permission. However, there is still no good way to ensure that. In this paper, we formulate the protection of released datasets as verifying whether they are adopted for training a (suspicious) third-party model, where defenders can only query the model while having no information about its parameters and training details. Based on this formulation, we propose to embed external patterns via backdoor watermarking for the ownership verification to protect them. Our method contains two main parts, including dataset watermarking and dataset verification. Specifically, we exploit poison-only backdoor attacks ($e.g.$, BadNets) for dataset watermarking and design a hypothesis-test-guided method for dataset verification. Experiments on multiple benchmark datasets of different tasks are conducted, which verify the effectiveness of our method. The code for reproducing main experiments is available at \url{https://github.com/THUYimingLi/DVBW}.
BibTeX
@article{li2023black,
  author = {Yiming Li and Mingyan Zhu and Xue Yang and Yong Jiang and Shu-Tao Xia},
  title = {{Black-box Ownership Verification for Dataset Protection via Backdoor Watermarking}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2023},
  doi = {10.1109/tifs.2023.3265535},
}
Figure