Robust Watermarking of Neural Network with Exponential Weighting

Ryota Namba, Jun Sakuma

TL;DR

This paper introduces a novel watermarking technique for neural networks that is robust against both model modification and query modification attacks. By combining a new "label change" key sample generation method with an "exponential weighting" embedding strategy, the authors achieve high verification accuracy and maintain predictive performance, addressing a critical need for intellectual property protection in deep learning.

What Problem Does This Paper Address?

Deep learning models are increasingly valuable intellectual property, but their distribution and use raise concerns about unauthorized redistribution or service provision. Digital watermarking offers a solution by embedding a verifiable ownership mechanism within the model. However, existing watermarking methods are vulnerable to attacks that attempt to remove or obscure the watermark. Specifically, attackers can modify the model itself (e.g., through pruning or retraining) or manipulate the queries used for verification. This paper tackles the challenge of creating a watermarking system that can withstand both types of attacks, ensuring reliable ownership verification even when faced with sophisticated adversaries.

Methods

The paper proposes a two-pronged approach:

  • Key Sample Generation (Label Change): Instead of relying on modified images (e.g., with superimposed logos or noise), this method uses original training samples but assigns them incorrect labels. This makes the key samples indistinguishable from normal training data to an unauthorized service provider, thwarting query modification attacks that rely on detecting unusual sample characteristics.
  • Embedding with Exponential Weighting: To make the watermark resilient to model modification (like pruning or retraining), the authors introduce an "exponential weighting" mechanism. This technique exponentially increases the influence of model parameters with large absolute values during training. This forces the model to rely heavily on these critical parameters, making it difficult for attackers to remove the watermark through modifications without significantly degrading the model's predictive performance.

Datasets

The authors evaluated their method on four standard image datasets:

  • MNIST
  • GTSRB (German Traffic Sign Recognition Benchmark)
  • CIFAR10
  • CIFAR100

Key Insights and Ideas

  • Query Modification is a Real Threat: The paper demonstrates that existing watermarking methods are vulnerable to "query modification," where an attacker uses techniques like autoencoders to alter queries and disrupt watermark verification.
  • Label Change is Stealthy: Using original samples with altered labels for key samples is a clever way to hide the watermark from unauthorized users, as these samples appear normal.
  • Exponential Weighting Fortifies the Watermark: By emphasizing high-magnitude weights, the proposed embedding method makes the watermark more robust to model modifications, preventing easy removal.
  • Complementary Attacks: The paper highlights that model modification and query modification attacks are often complementary; a method robust to one might be weak to the other. This underscores the need for a defense that addresses both.

Key Results

  • Robustness to Attacks: The proposed method achieved high watermark verification accuracy (AUC close to 1) even under combined model modification (pruning) and query modification (autoencoder) attacks across all tested datasets.
  • Preservation of Performance: The watermarking process resulted in only a minor decrease in the model's predictive performance (test accuracy), typically within 10%, making it practical for real-world deployment.
  • Superiority over Existing Methods: Compared to existing watermarking techniques, the proposed method demonstrated significantly better AUC under various attack scenarios, especially when the number of verification queries was limited. For instance, in Figure 2, the proposed method consistently achieved AUCs of 1.0, while existing methods often dropped significantly.

Limitations and Future Work

  • Hyperparameter Tuning: The effectiveness of the exponential weighting method relies on tuning the hyperparameter T. While the paper suggests T=2.0 worked well, optimal values might vary across different model architectures and datasets.
  • Computational Cost: While not explicitly detailed as a limitation, the exponential weighting mechanism and the training process for embedding might introduce additional computational overhead compared to simpler watermarking schemes.
  • Future Work: The authors suggest that further research could explore different activation functions for exponential weighting and investigate its application to other types of neural networks beyond image classification. They also implicitly suggest that exploring more sophisticated attack vectors could further validate the robustness of their proposed method.

Summary updated

Abstract

Deep learning has been achieving top levels of performance in many tasks. However, since it is costly to train a deep learning model, neural network models must be treated as valuable intellectual properties. One concern arising from our current situation is that malicious users might redistribute proprietary models or provide prediction services using such models without permission. One promising solution to this problem is digital watermarking, which works by embedding a mechanism into the model so that the model owners can verify their ownership of the model externally. In this study, we present a novel attack method against such watermarks known as query modification and demonstrate that all currently existing watermarking methods are vulnerable to either query modification or other existing attack methods (such as model modification). To overcome these vulnerabilities, we then present a novel watermarking method that we have named exponential weighting and experimentally show that our watermarking method achieves high watermark verification performance even under malicious invalidation processing attempts by unauthorized service providers (such as model modification and query modification) without sacrificing the predictive performance of the neural network model itself.
BibTeX
@inproceedings{namba2019robust,
  author = {Ryota Namba and Jun Sakuma},
  title = {{Robust Watermarking of Neural Network with Exponential Weighting}},
  booktitle = {AsiaCCS},
  year = {2019},
  doi = {10.1145/3321705.3329808},
}
Figure