FoldMark: Protecting Protein Generative Models with Watermarking

Zaixi Zhang, Ruofan Jin, Kaidi Fu, Le Cong, Marinka Zitnik, Mengdi WangarXiv 2024 · OpenReview

Abstract

Protein structure is key to understanding protein function and is essential for progress in bioengineering, drug discovery, and molecular biology. Recently, with the incorporation of generative AI, the power and accuracy of computational protein structure prediction/design have been improved significantly. However, ethical concerns such as copyright protection and harmful content generation (biosecurity) pose challenges to the wide implementation of protein generative models. Here, we investigate whether it is possible to embed watermarks into protein generative models and their outputs for copyright authentication and the tracking of generated structures. As a proof of concept, we propose a two-stage method FoldMark as a generalized watermarking strategy for protein generative models. FoldMark first pretrain watermark encoder and decoder, which can minorly adjust protein structures to embed user-specific information and faithfully recover the information from the encoded structure. In the second step, protein generative models are fine-tuned with watermark-conditioned Low-Rank Adaptation (LoRA) modules to preserve generation quality while learning to generate watermarked structures with high recovery rates. Extensive experiments are conducted on open-source protein structure prediction models (e.g., ESMFold and MultiFlow) and de novo structure design models (e.g., FrameDiff and FoldFlow) and we demonstrate that our method is effective across all these generative models. Meanwhile, our watermarking framework only exerts a negligible impact on the original protein structure quality and is robust under potential post-processing and adaptive attacks.

Introduction

The rapid advancement of generative AI in biology has revolutionized protein structure prediction and de novo design. However, this progress introduces significant risks regarding copyright infringement and biosecurity. Unlike text or images, protein structures are extremely sensitive to minor atomic shifts, and watermarks must be "geometry-aware" (SE(3)-equivariant) to remain detectable after spatial transformations. FoldMark addresses these challenges by embedding invisible, SE(3)-equivariant digital signatures into generated protein structures, allowing for reliable tracking without compromising biological integrity.

Methodology

FoldMark employs a two-stage strategy to integrate watermarks into protein generative models:

  1. Watermark Encoder/Decoder Pretraining: An SE(3)-equivariant encoder embeds a binary message ($m$) into a protein structure through minute adjustments to atomic coordinates. An invariant decoder is then trained to extract this message, ensuring the watermark is readable regardless of the protein's 3D orientation.
  2. Consistency-Preserving Finetuning (WaterLoRA): To adapt existing models like RFDiffusion or ESMFold, the authors introduce WaterLoRA (Watermark-conditioned Low-Rank Adaptation). This method uses small, trainable adapter layers modulated by a gating vector derived from the watermark code.
The WaterLoRA architecture enables consistency-preserving finetuning of generative models.

The training process optimizes two primary objectives: a Message Retrieval Loss for accurate decoding and a Consistency Loss to maintain the structural quality of the original model's outputs.

Experimental Setup

The framework was evaluated using a high-quality subset of the Protein Data Bank (PDB), consisting of 20,312 monomeric proteins (60–512 residues) with resolutions better than 5 Å. Proteins with excessive random loops (over 50%) were filtered out using DSSP. FoldMark was tested across diverse architectures, including prediction models (ESMFold, MultiFlow) and de novo design models (FrameDiff, FoldFlow).

Results and Discussion

FoldMark demonstrates exceptional performance in both watermark recovery and structural preservation.

  • High Fidelity: The system achieved nearly 100% bit accuracy for 4-bit to 16-bit codes. Even at 32 bits, accuracy remained robust (e.g., 85.0% for ESMFold).
  • Minimal Distortion: The structural impact is negligible, with RMSD increases typically staying below 0.1 Å, preserving the protein's biological utility.
  • Robustness: The watermarks are resilient to common modifications, including coordinate noise, random rotations, and sequence cropping (up to 50%).
Quantitative performance of FoldMark across different models and bit lengths.

By conditioning LoRA weights on the watermark code, FoldMark allows server owners to assign unique IDs to users, enabling precise traceability and identification within a pool of millions of users.

Conclusion and Future Work

FoldMark provides a robust solution for protecting the intellectual property and biosecurity of protein generative models. While effective, future work aims to address challenges related to large-scale conformational changes and end-to-end training pipelines. Additionally, optimizing the technique for complex tasks like motif scaffolding remains an area for further development.

Summary updated

Abstract

Protein structure is key to understanding protein function and is essential for progress in bioengineering, drug discovery, and molecular biology. Recently, with the incorporation of generative AI, the power and accuracy of computational protein structure prediction/design have been improved significantly. However, ethical concerns such as copyright protection and harmful content generation (biosecurity) pose challenges to the wide implementation of protein generative models. Here, we investigate whether it is possible to embed watermarks into protein generative models and their outputs for copyright authentication and the tracking of generated structures. As a proof of concept, we propose a two-stage method FoldMark as a generalized watermarking strategy for protein generative models. FoldMark first pretrain watermark encoder and decoder, which can minorly adjust protein structures to embed user-specific information and faithfully recover the information from the encoded structure. In the second step, protein generative models are fine-tuned with watermark-conditioned Low-Rank Adaptation (LoRA) modules to preserve generation quality while learning to generate watermarked structures with high recovery rates. Extensive experiments are conducted on open-source protein structure prediction models (e.g., ESMFold and MultiFlow) and de novo structure design models (e.g., FrameDiff and FoldFlow) and we demonstrate that our method is effective across all these generative models. Meanwhile, our watermarking framework only exerts a negligible impact on the original protein structure quality and is robust under potential post-processing and adaptive attacks.
BibTeX
@misc{zhang2024foldmark,
  author = {Zaixi Zhang and Ruofan Jin and Kaidi Fu and Le Cong and Marinka Zitnik and Mengdi Wang},
  title = {{FoldMark: Protecting Protein Generative Models with Watermarking}},
  year = {2024},
  eprint = {2410.20354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.20354},
}
Figure