DeepMarks: A Digital Fingerprinting Framework for Deep Neural Networks

Huili Chen, Bita Darvish Rouhani, Farinaz Koushanfar

TL;DR

This paper introduces DeepMarks, a novel framework for digitally fingerprinting deep neural networks (DNNs) to protect intellectual property and track unauthorized usage. It embeds unique, user-specific fingerprints within the DNN's weights, allowing owners to identify who is using their models and how, even against sophisticated attacks like model compression and fine-tuning.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of deep learning have led to a booming trend of sharing pre-trained DNN models. While this democratizes access to powerful AI tools, it also creates significant challenges for model owners regarding intellectual property (IP) protection and digital rights management (DRM). Training these models requires substantial computational resources and data, making them valuable assets. Without effective protection, malicious actors can easily misuse or redistribute these models, undermining the owner's competitive advantage. Existing methods for protecting digital content, like watermarking, haven't been comprehensively adapted to the unique characteristics of DNNs, especially concerning the robustness required against model transformations and collusion among users.

Methods

DeepMarks proposes an end-to-end framework for embedding and extracting digital fingerprints within DNNs. The core idea is to leverage the inherent "extra capacity" within DNNs to encode a unique binary code-vector (fingerprint) for each user. This fingerprint is embedded in the probability density function (pdf) of the trainable weights.

The framework offers two primary fingerprinting methodologies:

  • Orthogonal Fingerprinting: This method assigns a unique orthogonal vector to each user. It's simple to implement and effective for distinguishing individual users. However, it's less resilient to collusion attacks.
  • Coded Fingerprinting: This approach uses anti-collusion codes (ACC), specifically leveraging Balanced Incomplete Block Designs (BIBD), to create more robust fingerprints. BIBDs allow for a larger number of users than the fingerprint dimension and are designed to resist collusion attacks where multiple users combine their models. This method is more complex but offers superior resilience.

The embedding process involves adding a specific loss term to the standard training or fine-tuning objective function, encouraging the model's weights to conform to the desired fingerprint distribution. Extraction involves analyzing the weights of the marked layers to recover the embedded fingerprint and identify the user.

Datasets

The paper evaluates DeepMarks on two widely used benchmark datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR10: A dataset of small color images in 10 classes.

Key Insights and Ideas

  • DNNs as a Canvas for Fingerprinting: The research demonstrates that the internal parameters (weights) of DNNs can be effectively utilized to embed unique, robust digital fingerprints without significantly degrading model performance.
  • Coded Fingerprinting for Collusion Resistance: The adoption of BIBD-based anti-collusion codes is a crucial innovation, enabling the framework to withstand attacks where multiple users collaborate to remove or obscure fingerprints.
  • Generality Across Architectures: DeepMarks is shown to be effective across different DNN architectures (CNNs and WRNs) and datasets, highlighting its broad applicability.
  • Robustness to Common Attacks: The framework exhibits resilience against common model manipulation techniques like parameter pruning and model fine-tuning, which are often used to remove watermarks.

Key Results

  • Fidelity: DeepMarks embedding resulted in no significant drop in test accuracy, and in some cases, even slightly improved it compared to fine-tuned models without fingerprints. For example, on MNIST-CNN, baseline accuracy was 99.52%, fine-tuned without fingerprint was 99.66%, and fine-tuned with fingerprint was 99.72%.
  • Collusion Resilience: Using a (31, 6, 1)-BIBD ACC codebook, DeepMarks achieved 100% detection rate for up to 5 colluders (Kmax = 5) with a 0% false alarm rate. This resilience was consistent across MNIST-CNN and CIFAR10-WRN benchmarks.
  • Robustness to Fine-tuning: The framework maintained its collusion resistance level (Kmax = 5) even when the marked neural network was fine-tuned.
  • Robustness to Pruning: DeepMarks demonstrated 100% decoding accuracy for extracted code-vectors even with pruning rates up to 99%. The detection rates for collusion attacks remained robust, maintaining Kmax = 5.
  • Efficiency: Embedding fingerprints required retraining for as few as 5 epochs, suggesting a negligible runtime overhead.

Limitations and Future Work

  • Orthogonal Fingerprinting Limitations: While simple, orthogonal fingerprinting was found to be less scalable and susceptible to collusion attacks compared to coded fingerprinting. The paper notes that it's a special case of coded fingerprinting.
  • Colluder Detection Threshold: For orthogonal fingerprinting, the paper highlights that there's no single decision threshold that can guarantee complete detection of all colluders without also leading to false positives.
  • Future Work: The paper suggests that by expanding the ACC codebook and choosing appropriate parameters, DeepMarks can support an increasingly large number of users, ensuring scalability for widespread model distribution. The generic nature of the framework also implies potential applicability to other machine learning models beyond DNNs.

Summary updated

Abstract

This paper proposes DeepMarks, a novel end-to-end framework for systematic fingerprinting in the context of Deep Learning (DL). Remarkable progress has been made in the area of deep learning. Sharing the trained DL models has become a trend that is ubiquitous in various fields ranging from biomedical diagnosis to stock prediction. As the availability and popularity of pre-trained models are increasing, it is critical to protect the Intellectual Property (IP) of the model owner. DeepMarks introduces the first fingerprinting methodology that enables the model owner to embed unique fingerprints within the parameters (weights) of her model and later identify undesired usages of her distributed models. The proposed framework embeds the fingerprints in the Probability Density Function (pdf) of trainable weights by leveraging the extra capacity available in contemporary DL models. DeepMarks is robust against fingerprints collusion as well as network transformation attacks, including model compression and model fine-tuning. Extensive proof-of-concept evaluations on MNIST and CIFAR10 datasets, as well as a wide variety of deep neural networks architectures such as Wide Residual Networks (WRNs) and Convolutional Neural Networks (CNNs), corroborate the effectiveness and robustness of DeepMarks framework.
BibTeX
@article{chen2018deepmarks,
  author = {Huili Chen and Bita Darvish Rouhani and Farinaz Koushanfar},
  title = {{DeepMarks: A Digital Fingerprinting Framework for Deep Neural Networks}},
  journal = {IACR Cryptol. ePrint Arch.},
  year = {2018},
  url = {https://openreview.net/forum?id=_ZHPe_nTHM2},
}
Figure