Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable

Haozhe Liu, Wentian Zhang, Bing Li, Bernard Ghanem, Jürgen SchmidhuberarXiv 2024 · DOI · arXiv

Abstract

The rapid proliferation of foundational diffusion models (DMs) has intensified the need for robust intellectual property (IP) protection. While backdoor watermarking offers a potential solution, such identifiers are frequently erased during downstream fine-tuning. This work identifies a Pareto-like distribution in model updates, where a small subset of "busy" layers drives performance gains while "lazy" layers remain largely unchanged. We introduce the Arbitrary-In—Arbitrary-Out (AIAO) strategy, which embeds watermarks within these stable lazy layers. By utilizing subpath embedding and feature-space sign flipping, our method maintains high verification rates across diverse fine-tuning scenarios with negligible impact on image fidelity.

Introduction

Foundational diffusion models like Stable Diffusion have revolutionized digital art and content creation. However, model creators face significant challenges in protecting their IP. A common technique is backdoor watermarking, where a model is trained to generate a specific trigger response upon receiving a secret input. Unfortunately, these watermarks are often "forgotten" when users fine-tune the model for specific tasks, a process known as fine-tuning-based removal. To address this, we propose a method that targets the inherent structural stability of specific layers within the DM architecture.

The "Lazy Layer" Phenomenon

Our investigation into the fine-tuning process reveals a critical dichotomy in layer behavior. We observe that approximately 80% of the performance improvement during fine-tuning is attributed to only 20% of the model's layers, which we term "busy layers." Conversely, the remaining "lazy layers" undergo minimal parameter shifts and contribute significantly less to the new task's performance.

Figure 2: Distribution of parameter differences during fine-tuning, highlighting the distinction between "lazy" layers and "busy" layers.

These lazy layers represent an ideal location for embedding persistent watermarks. Because they are not actively utilized for learning new data distributions, they are less likely to be overwritten during subsequent fine-tuning stages, effectively acting as a form of "continual learning" for the watermark.

Methodology: The AIAO Strategy

To exploit the stability of lazy layers, we propose the Arbitrary-In—Arbitrary-Out (AIAO) strategy. This approach moves beyond global model watermarking by focusing on internal sub-networks.

Subpath Embedding and Trigger Functions

AIAO randomly samples pairs of layers—an "in-layer" and an "out-layer"-to define a subpath. The watermark is embedded by training this sub-network to produce a predefined response. Crucially, we employ a feature-space sign flipping trigger. Instead of altering pixel values or feature magnitudes, we use a mask-controlled function to flip the signs (positive/negative) of specific elements in the internal feature maps. This "shape-free" trigger ensures the watermark remains invisible and does not interfere with the model's primary generative capabilities.

Figure 4: The AIAO framework, illustrating subpath selection, the sign-flipping trigger function, and the verification process.

Monte Carlo Verification

Ownership is verified through Monte Carlo sampling. By checking the average response across multiple random subpaths during image generation, we can reliably confirm the presence of the watermark even if some individual paths have been slightly altered by fine-tuning.

Experimental Evaluation

We evaluated AIAO across several benchmarks, including MS-COCO for general generation and AFHQ/CUB-200 for specialized fine-tuning. Our results demonstrate that AIAO maintains a verification rate consistently above 90%, even after extensive fine-tuning that typically degrades traditional watermarks to ~70%. Furthermore, standard metrics such as Fréchet Inception Distance (FID) and CLIP scores show that AIAO-watermarked models perform identically to their un-watermarked counterparts, ensuring no trade-off between traceability and image quality.

Conclusion

By identifying and leveraging "lazy layers," we provide a more resilient method for tracing the lineage of fine-tuned diffusion models. The AIAO strategy offers a robust defense against IP theft in the open-weights ecosystem. Future work will explore extending these internal feature-space triggers to black-box verification scenarios where internal model weights are not directly accessible.

Summary updated

Abstract

Foundational generative models should be traceable to protect their owners and facilitate safety regulation. To achieve this, traditional approaches embed identifiers based on supervisory trigger-response signals, which are commonly known as backdoor watermarks. They are prone to failure when the model is fine-tuned with nontrigger data. Our experiments show that this vulnerability is due to energetic changes in only a few 'busy' layers during fine-tuning. This yields a novel arbitrary-in-arbitrary-out (AIAO) strategy that makes watermarks resilient to fine-tuning-based removal. The trigger-response pairs of AIAO samples across various neural network depths can be used to construct watermarked subpaths, employing Monte Carlo sampling to achieve stable verification results. In addition, unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths, where a mask-controlled trigger function is proposed to preserve the generation performance and ensure the invisibility of the embedded backdoor. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO; while the verification rates of other trigger-based methods fall from ~90% to ~70% after fine-tuning, those of our method remain consistently above 90%.
BibTeX
@misc{liu2024lazy,
  author = {Haozhe Liu and Wentian Zhang and Bing Li and Bernard Ghanem and Jürgen Schmidhuber},
  title = {{Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable}},
  year = {2024},
  eprint = {2405.00466},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.00466},
}
Figure