Structural Watermarking to Deep Neural Networks via Network Channel Pruning

Xiangyu Zhao, Yinzhe Yao, Hanzhou Wu, Xinpeng Zhang

Introduction

Training high-performance Deep Neural Networks (DNNs) is a resource-intensive process, requiring massive datasets, specialized expertise, and significant computational power. Consequently, these models represent valuable intellectual property (IP) that necessitates robust protection against unauthorized redistribution.

Traditional watermarking techniques typically rely on parameter-based methods, either embedding data into weight distributions (white-box) or triggering specific "backdoor" behaviors (black-box). However, because these methods depend on specific weight values, they are highly vulnerable to "removal attacks." Adversaries can often eliminate these watermarks through fine-tuning on small datasets or standard model compression techniques that alter the network's weights.

Structural Watermarking via Channel Pruning

To address these vulnerabilities, this paper proposes a "structural watermarking" scheme. Instead of modifying weights, ownership information is embedded directly into the network's topology. The core idea is that the number of channels in a layer is a "hard" architectural property, making it significantly more difficult to alter without compromising the model's utility.

The proposed method leverages Network Channel Pruning as the embedding mechanism:

  1. Watermark Partitioning: A secret bit-sequence is divided into segments.
  2. Mapping to Pruning Rates: Each segment is mapped to a specific pruning rate (the percentage of channels removed in a layer) using Quantization Index Modulation (QIM).
  3. Structural Embedding: The host DNN is pruned layer-by-layer according to these rates. The authors evaluated two strategies: Network Slimming (based on Batch Normalization scaling factors) and L1-norm Pruning (based on absolute weight values).
  4. Fine-tuning: The pruned model is fine-tuned to recover accuracy lost during the structural change.
  5. Extraction: Ownership is verified by comparing the marked model's channel counts against the original architecture to reconstruct the bit-sequence.

Experimental Evaluation and Results

The researchers evaluated the scheme using the CIFAR-10 and CIFAR-100 benchmarks across three architectures: VGG-19, ResNet-164, and DenseNet-40.

High Fidelity

A critical requirement for any watermarking scheme is maintaining the original model's performance. The structural approach demonstrates exceptional fidelity. For instance, a VGG-19 model on CIFAR-10 experienced only a negligible drop in precision (from 92.70% to approximately 92.64%) after embedding the watermark.

Precision vs. embedding rate ($r_{cov}$) for VGG-19 on CIFAR-10, showing minimal performance impact.

As shown in the figure above, the precision remains remarkably stable even as the embedding coverage ($r_{cov}$) increases, regardless of the pruning strategy used. This stability allows for reliable ownership verification without sacrificing the model's primary utility.

Robustness and Capacity

The scheme is inherently robust against parameter-altering attacks. Since the watermark is stored in the count of channels per layer, it remains intact even if an attacker fine-tunes the weights or applies weight re-quantization. The capacity of the watermark scales with network depth; deeper models like ResNet-164 can hide hundreds of bits.

Comparison of pruning strategies across varying embedding rates for a model on CIFAR-100.

While both pruning strategies maintain high precision, the choice of method can influence the trade-off between embedding rate and performance, as seen in the results for more complex datasets like CIFAR-100 where the baseline precision is lower.

Conclusion

Structural watermarking provides a powerful alternative to traditional weight-based methods. By tying the watermark to the network's architecture, the authors create a protection mechanism that is resistant to common removal attacks while maintaining high model accuracy. Future work aims to develop "blind" extraction systems and further investigate defenses against structural attacks like knowledge distillation.

Summary updated

BibTeX
@inproceedings{zhao2021structural,
  author = {Xiangyu Zhao and Yinzhe Yao and Hanzhou Wu and Xinpeng Zhang},
  title = {{Structural Watermarking to Deep Neural Networks via Network Channel Pruning}},
  booktitle = {WIFS},
  year = {2021},
  doi = {10.1109/wifs53200.2021.9648376},
}
Figure