A deployed model that answers queries through a paid interface can be copied. An attacker sends it many inputs, records the answers, and trains a cheap imitation. The strongest forensic answer to this is the entangled watermark. The owner trains the model so that a few secret trigger inputs share internal features with ordinary inputs, so any copy that learns the real task also inherits the secret behavior. That guarantee was established for a model trained once and then frozen. Commercial models are not frozen; they keep learning new classes from new data. The method an owner would use to update them, continual learning (training on a stream of new tasks while a loss term keeps old behavior from being overwritten), was never designed to preserve the property the watermark depends on: that triggers and real classes stay tangled together inside the network.
This note claims that two things come apart during continual updates. One is whether the model itself still responds to its triggers. The other is whether a copy stolen from it inherits them. Replaying the triggers in the memory buffer protects the first and not the second. We propose to make the entanglement itself a replayed objective. During each update, a representation-level penalty is applied to mixed batches of buffered old-class samples, new-class samples, and triggers. We test whether copies stolen at any point in the stream still carry a verifiable watermark. In two sentences: an owner who keeps updating a watermarked model will quietly lose the ability to prove theft unless the update loss explicitly keeps triggers entangled with both old and new classes. We build that loss and measure the property on copies, not on the victim.
What prior work establishes and leaves open
The watermark mechanism comes from Entangled Watermarks as a Defense against Model Extraction. That paper adds a Soft Nearest Neighbor Loss, a penalty that pulls an example's intermediate features toward those of examples with a different label and so mixes classes in feature space. The penalty makes triggers share features with a real class. Its key finding motivates this proposal. Ordinary watermarks, which the victim model reproduces perfectly, survive in stolen copies at under 10%. Entangled ones survive at 38% on average (up to 65% on MNIST), at a cost of about 0.81% test accuracy. So the victim knowing its watermark is not enough; what transfers is shared representation. The paper evaluates only static, single-session training on MNIST, Fashion-MNIST and CIFAR-10.
The update machinery comes from the dark-experience line. Dark Experience for General Continual Learning stores past inputs together with the model's output scores at the time and penalizes drift from those scores (DER, and DER++ which also replays labels). Class-Incremental Continual Learning Into the eXtended DER-Verse adds X-DER, which revises stored scores as new classes arrive. Both constrain outputs on buffered points. Neither says anything about the relative geometry of features for points the buffer underrepresents, such as triggers. Adding new class prototypes is exactly the kind of change that shifts the distances the entanglement penalty relies on.
The threat model is constrained by two papers. Knockoff Nets shows that copies built from generic out-of-distribution queries recover 81% to 96% of victim accuracy. The attacker therefore probes regions that neither the continual-learning loss nor the buffer covers. Attention Distraction is the closest bridging work, and it points the other way from us. It uses continual learning with selective forgetting as an attack: a ten-image "lure" task plus anchoring on proxy data drives watermark accuracy below 20%, often near 0%, with under 2% main-task loss. Its authors leave open whether entangled watermarks resist this.
Our proposal reverses that direction: the defender, not the attacker, controls a continual-learning loss and uses it to keep the watermark. Learning with Selective Forgetting shows that synthetic class-bound codes can be kept or dropped inside a composite continual loss. Its codes are internal memory devices that were never tested against an external thief. Continual Adversarial Defense makes a defense continual, but against input-perturbation attacks rather than theft.
Core Claim
Under class-incremental updates, the watermark success rate in copies extracted from version $t$ will track how entangled the triggers are in version $t$. It will not track how well version $t$ itself responds to triggers. Standard rehearsal, even with triggers placed in the buffer, preserves the second and lets the first decay. A replayed entanglement objective preserves both, at a measurable plasticity cost that we expect to be small because the dark-experience losses already favor flat minima.
Research Program
Version 0 is trained with the entangled-watermark recipe: triggers derived from a source class, labeled as a target class, with the Soft Nearest Neighbor penalty. Each later task is learned with DER++ or X-DER plus one added term:
$$\mathcal{L} = \mathcal{L}_{\text{DER++}} + \lambda\,\mathcal{L}_{\text{SNN}}\big(B_{\text{new}} \cup B_{\mathcal{M}} \cup W\big),$$
Here $\mathcal{L}_{\text{DER++}}$ is the usual task-plus-replay loss. $\mathcal{L}_{\text{SNN}}$ is the Soft Nearest Neighbor entanglement term from the entangled-watermark paper, computed at the same intermediate layers. $B_{\text{new}}$ is the current batch, $B_{\mathcal{M}}$ is a batch drawn from the replay buffer $\mathcal{M}$, $W$ is a small fixed set of stored triggers, and $\lambda$ sets the weight.
The essential choice is that the penalty is computed over old and new classes together with triggers. The entanglement is then re-established against the whole current class geometry, not only against task-0 classes. Triggers also enter the buffer with their stored scores, so the X-DER score-revision step keeps them consistent with new heads.
We also track a cheap diagnostic at every version: the Soft Nearest Neighbor value between triggers and real classes. This tests the central prediction that the diagnostic, rather than victim trigger accuracy, predicts what copies inherit.
Baselines
The naive reference point trains the entangled watermark at version 0 and then updates with plain DER++ or X-DER, ignoring the watermark. This is what an owner would do today, and it is the condition we expect to decay.
The strongest simple competitor is trigger replay. Triggers are stored in the buffer with their scores and rehearsed by the dark-experience loss, with no entanglement term during updates. It costs almost nothing and keeps the victim's trigger behavior intact, so it is the alternative most likely to make our added term unnecessary.
The offline upper bound re-runs the full entangled-watermark training on all data seen so far at each version. It shows how much transfer a continual method could hope to keep, though it violates the no-stored-data constraint that motivates continual learning.
A forensic alternative that needs no training change uses conferrable adversarial-example fingerprints, from Deep Neural Network Fingerprinting by Conferrable Adversarial Examples, recomputed for each released version. We found no reported result on Lacuna for this fingerprinting method on Split CIFAR-100 or under continual updates.
Two ablations isolate the key ingredient. One computes the entanglement term only on the current batch plus triggers, dropping buffered old-class samples. The other drops X-DER score revision for triggers.
All methods share the backbone (a ResNet-18 as used in the dark-experience benchmarks), buffer size, number of epochs and a fixed hyperparameter search budget. Buffer slots spent on triggers count against the same total memory.
For continual accuracy, the reference numbers on Split CIFAR-100 with a 500-sample buffer are from the X-DER paper: X-DER reaches 49.93% final average accuracy with 19.90% final forgetting, against DER++ at 38.25% and 50.54%. On Sequential CIFAR-10 with a 200-sample buffer, DER++ reaches 64.88% in the DER paper. For copy-side watermark transfer, the reference is the entangled-watermark paper's static result of 38% average retention versus under 10% for ordinary watermarks, with ownership established at 95% confidence in under 100 queries.
Beating trigger replay requires three things together. Copies extracted from versions 3 onward must show clearly higher watermark success than under trigger replay. That success must stay within reach of the offline upper bound and of the static 38% figure. And the final average accuracy cost relative to unmodified X-DER must be no more than about one point, mirroring the static 0.81% cost.
Why not just…?
Why not just put the triggers in the replay buffer? This is the real threat, and the experiment is built around it. The entangled-watermark paper is the evidence that it should fall short: watermarks the victim reproduces perfectly still transfer to copies under 10% of the time when they occupy separate features. Trigger replay protects the victim's outputs, not the shared features. If copies from trigger-replay models nonetheless match our method, the representation term is unnecessary. We would then report that cheap rehearsal suffices, a useful negative result, rather than defend the added loss.
Why not just fingerprint each released version? Fingerprints need no training change, which is attractive. They are tied to the decision boundary of one specific version, and that boundary drifts at every update. The owner must therefore store and test one fingerprint per version without knowing which version was copied, and copies from older versions may match no current fingerprint. The comparison measures exactly this: verification success and query cost when the thief's source version is unknown.
Why not just retrain the watermark from scratch at each update? That is our offline upper bound, and it is the right ceiling. It requires keeping all past training data, the constraint that continual learning exists to remove, and it costs a full entangled-watermark training run per version (1.5x to 2x training overhead per the entangled-watermark paper). The gap between it and our method is the price of staying continual.
Proposed Experiments
The main benchmark is Split CIFAR-100 in ten tasks, with Sequential CIFAR-10 as a smaller check and Split miniImageNet in twenty tasks for scale. After every task, we extract a copy with the Knockoff Nets procedure. Two query pools are used: generic out-of-distribution images (TinyImageNet or an ImageNet subset) and in-distribution held-out data, the harder case. We record the victim's accuracy, the copy's accuracy, the copy's watermark success rate, and the number of queries needed for 95%-confidence ownership verification using the entangled-watermark statistical test.
The first decisive check is a correlation across all versions and methods. Copy watermark success should track the trigger–class entanglement diagnostic and should not track victim trigger accuracy. If victim accuracy predicts it equally well, the core claim fails.
The second check is the curve of copy watermark success against task index for naive, trigger-replay and our method. Our method should hold roughly flat while the others decline.
The third check takes the final model of each method and attacks it with Attention Distraction using its published recipe of ten lure images and proxy data. The test is whether continual entanglement keeps watermark accuracy above the below-20% level that attack achieves on non-entangled watermarks, while the attack's main-task loss is held to its reported under-2% margin. This directly answers the open question that paper poses about entangled watermarks.
Finally, plasticity cost is read off final average accuracy and final forgetting against the X-DER and DER++ figures above, and we sweep $\lambda$ to trace the trade-off between watermark transfer and accuracy.
References
- Entangled Watermarks as a Defense against Model Extraction
- Dark Experience for General Continual Learning: a Strong, Simple Baseline
- Class-Incremental Continual Learning into the eXtended DER-verse
- Knockoff Nets: Stealing Functionality of Black-Box Models
- Attention Distraction: Watermark Removal Through Continual Learning with Selective Forgetting
- Learning with Selective Forgetting
- Continual Adversarial Defense
- Deep Neural Network Fingerprinting by Conferrable Adversarial Examples