Training state-of-the-art deep learning models is an expensive endeavor, requiring massive datasets and significant computational power. This makes high-performing models valuable intellectual property (IP). However, they are increasingly vulnerable to "model stealing" or "extraction attacks," where an adversary queries a model's API to train a surrogate model that mimics the original's functionality.
To counter this, researchers have developed a defense strategy based on watermark inheritance. Instead of trying to prevent the theft, this approach ensures that any stolen model inadvertently carries a hidden, verifiable signature from the original.
The Intuition of Behavioral Mimicry
The core idea behind watermark inheritance relies on the very mechanism of model stealing. In a typical extraction attack, an adversary treats the target model as a "teacher" and trains a "student" (surrogate) model to minimize the divergence between their outputs.
As Jie Zhang and Dongdong Chen argue in Deep Model Intellectual Property Protection via Deep Watermarking, neural networks are universal approximators that naturally learn to reproduce any consistent signal present in their training data. If a teacher model’s outputs are subtly "poisoned" with a hidden watermark, the student model will treat this signal as part of the ground truth distribution. Consequently, the more effectively an attacker mimics the target model's functionality, the more deeply they embed the hidden ownership signature into the stolen model.
Embedding the Radioactive Tracer
For a watermark to be effective, it must be "radioactive"—meaning it contaminates any derivative model trained on it. Researchers use several techniques to inject these tracers into the model's behavior:
- Logit Perturbation: Methods like DRW (Distillation-Resistant Watermarking) by Lei Li and Xuandong Zhao inject a subtle sinusoidal signal into the model’s output probability distributions (Distillation-Resistant Watermarking for Model Protection in NLP). Because student models learn to mimic these raw scores (logits) during distillation, they inherit the periodic signal.
- Out-of-Distribution (OoD) Triggers: Shuyang Yu demonstrates that watermarks can be injected using a single OoD image, creating a specific response that persists even when the student model is trained on a different dataset (Safe and Robust Watermark Injection with a Single OoD Image).
- Structural Permutation: Hengyuan Xu and Liyao Xiang propose TokenMark, which exploits the permutation equivariance of Transformers to embed structural signatures that are inherited during extraction (TokenMark: A Modality-Agnostic Watermark for Pre-trained Transformers).
Why Inheritance is Robust
A critical challenge for IP protection is that an attacker might use a different neural architecture or a different optimization method (like LoRA vs. full fine-tuning) for their stolen model.
However, evidence suggests that watermark inheritance is largely cross-architectural. As noted by Mikhail Pautov and Oleg Rogov, watermarks based on stable decision regions persist because certain learned features and decision boundaries are fundamental enough to be captured by diverse model structures during the distillation process (Probabilistically Robust Watermarking of Neural Networks).
To further ensure this robustness, researchers like Sarthak Kumar Maharana suggest aligning watermark triggers with internal feature representations rather than arbitrary labels. In the MAT framework, the watermark becomes a functional part of the model's logic, making it nearly impossible to remove without destroying the model's primary utility (Not Just Change the Labels, Learn the Features: Watermarking Deep Neural Networks with Multi-view Data).
Verification in the Black Box
The ultimate goal of these methods is post-hoc ownership verification. If a company suspects a competitor has stolen their model, they can query the suspicious API with specific "trigger" inputs.
If the suspicious model was indeed trained via extraction from the watermarked source, it will exhibit the inherited behavior—such as specific synonym swaps in text (Qiongkai Xu et al., Protecting Intellectual Property of Language Generation APIs with Lexical Watermark) or specific artifacts in generated images (Hailong Hu and Jun Pang 0001, Ownership Protection of Generative Adversarial Networks). This provides a statistically significant "fingerprint" that serves as undeniable proof of intellectual property theft.