Abstract
The rapid proliferation of foundational diffusion models (DMs) has intensified the need for robust intellectual property (IP) protection. While backdoor watermarking offers a potential solution, such identifiers are frequently erased during downstream fine-tuning. This work identifies a Pareto-like distribution in model updates, where a small subset of "busy" layers drives performance gains while "lazy" layers remain largely unchanged. We introduce the Arbitrary-In—Arbitrary-Out (AIAO) strategy, which embeds watermarks within these stable lazy layers. By utilizing subpath embedding and feature-space sign flipping, our method maintains high verification rates across diverse fine-tuning scenarios with negligible impact on image fidelity.
Introduction
Foundational diffusion models like Stable Diffusion have revolutionized digital art and content creation. However, model creators face significant challenges in protecting their IP. A common technique is backdoor watermarking, where a model is trained to generate a specific trigger response upon receiving a secret input. Unfortunately, these watermarks are often "forgotten" when users fine-tune the model for specific tasks, a process known as fine-tuning-based removal. To address this, we propose a method that targets the inherent structural stability of specific layers within the DM architecture.
The "Lazy Layer" Phenomenon
Our investigation into the fine-tuning process reveals a critical dichotomy in layer behavior. We observe that approximately 80% of the performance improvement during fine-tuning is attributed to only 20% of the model's layers, which we term "busy layers." Conversely, the remaining "lazy layers" undergo minimal parameter shifts and contribute significantly less to the new task's performance.
These lazy layers represent an ideal location for embedding persistent watermarks. Because they are not actively utilized for learning new data distributions, they are less likely to be overwritten during subsequent fine-tuning stages, effectively acting as a form of "continual learning" for the watermark.
Methodology: The AIAO Strategy
To exploit the stability of lazy layers, we propose the Arbitrary-In—Arbitrary-Out (AIAO) strategy. This approach moves beyond global model watermarking by focusing on internal sub-networks.
Subpath Embedding and Trigger Functions
AIAO randomly samples pairs of layers—an "in-layer" and an "out-layer"-to define a subpath. The watermark is embedded by training this sub-network to produce a predefined response. Crucially, we employ a feature-space sign flipping trigger. Instead of altering pixel values or feature magnitudes, we use a mask-controlled function to flip the signs (positive/negative) of specific elements in the internal feature maps. This "shape-free" trigger ensures the watermark remains invisible and does not interfere with the model's primary generative capabilities.
Monte Carlo Verification
Ownership is verified through Monte Carlo sampling. By checking the average response across multiple random subpaths during image generation, we can reliably confirm the presence of the watermark even if some individual paths have been slightly altered by fine-tuning.
Experimental Evaluation
We evaluated AIAO across several benchmarks, including MS-COCO for general generation and AFHQ/CUB-200 for specialized fine-tuning. Our results demonstrate that AIAO maintains a verification rate consistently above 90%, even after extensive fine-tuning that typically degrades traditional watermarks to ~70%. Furthermore, standard metrics such as Fréchet Inception Distance (FID) and CLIP scores show that AIAO-watermarked models perform identically to their un-watermarked counterparts, ensuring no trade-off between traceability and image quality.
Conclusion
By identifying and leveraging "lazy layers," we provide a more resilient method for tracing the lineage of fine-tuned diffusion models. The AIAO strategy offers a robust defense against IP theft in the open-weights ecosystem. Future work will explore extending these internal feature-space triggers to black-box verification scenarios where internal model weights are not directly accessible.
Summary updated