Introduction
The Lottery Ticket Hypothesis (LTH) suggests that dense neural networks contain sparse subnetworks—often called "winning tickets"—that can match the performance of the original dense model when trained in isolation from the same initialization. However, finding these tickets typically requires Iterative Magnitude Pruning (IMP), a computationally expensive process involving multiple rounds of training and pruning.
Because of this high search cost, a discovered winning ticket is a valuable proprietary asset. Standard model watermarking techniques usually focus on modifying weights or training on "trigger sets" (backdoor-style inputs). Unfortunately, these methods do not specifically protect the sparsity pattern (the mask) itself. If an attacker steals the mask, they can simply train the subnetwork from scratch. This paper addresses the gap in Intellectual Property (IP) protection by embedding ownership signatures directly into the network's sparse topology, allowing winning tickets to claim their own ownership while remaining robust against fine-tuning, pruning, and ambiguity attacks.
Methods for Lottery Verification
The authors propose three verification schemes based on sparse structural information to protect these valuable masks:
- Scheme $V_1$ (Key-Lock Splitting): The winning ticket's mask is split into two parts: a "locked mask" and a "key mask." Without the key mask, the model's performance degrades significantly. Legitimate users must combine both to recover the full winning ticket.
- Scheme $V_2$ (Signature Embedding): This is a "white-box" method where a signature is embedded directly into the binary sparsity pattern of the model's low-level layers. The authors use a modified QR code (removing the finder and alignment patterns to make it imperceptible) as the credential.
- Scheme $V_3$ (Trigger Set Integration): To enable "black-box" verification (where the owner only has API access to the model), the authors combine mask embedding with trigger-set training. The model is trained to produce specific, incorrect labels for a secret set of "trigger" images, acting as a digital fingerprint.
The core technical novelty lies in an algorithm that identifies regions in the sparse mask most similar to the desired signature and "absorbs" the signature into the mask with minimal impact on accuracy.
Key Insights
- Sparsity as a Credential: Unlike dense models, "extreme" winning tickets are naturally robust to further pruning. If you fine-tune them, the weights change but the topology (the mask) stays the same. This makes the mask an ideal place to hide a signature.
- QR Codes are Resilient: The authors chose QR codes because of their inherent error-correction capabilities. Even if an attacker prunes 20% of the model or adds noise to the mask, the owner can often still decode the signature.
- The "Add-on" Attack Defense: The authors introduce a new type of attack where an adversary tries to "contaminate" the mask by recovering pruned weights or adding noise. They show that their defense can filter out low-magnitude noise to maintain signature integrity.
- Low-Level Layers are Best: Information embedded in earlier convolutional layers is harder to remove because these layers are generally less sparse and more critical to feature extraction than deeper layers.
Key Results
The researchers evaluated their methods using standard computer vision benchmarks (CIFAR-10 and CIFAR-100) across several architectures (ResNet-20, ResNet-18, and ResNet-50).
- Minimal Performance Hit: Embedding a QR code signature resulted in negligible accuracy drops. For example, ResNet-20 on CIFAR-10 saw a drop of only 0.29%.
- Robustness to Pruning: In ResNet-20, the embedded QR code remained decodable even after an additional 20% of the weights were pruned using one-shot magnitude pruning.
- Black-Box Success: Under Scheme $V_3$, the model maintained high accuracy on the main task while achieving over 90% detection accuracy on the secret trigger set.
- Transferability Defense: The "Key-Lock" method ($V_1$) proved highly effective. If an attacker tried to use a stolen "locked" mask on a different dataset, the performance gap remained large (over 3%), proving the mask's specificity to the original task and initialization.
Limitations and Future Work
While the paper successfully protects the result of the search, it doesn't reduce the initial high cost of finding the winning ticket itself. Furthermore, the study focuses primarily on ResNets and convolutional layers; extending this to other architectures, like Transformers or GNNs, is a logical next step. Finally, while the methods resist current "ambiguity attacks," a sophisticated "insider" who knows exactly where the signature is hidden might still attempt to overwrite it, though this would likely degrade model performance significantly.
Summary updated