TL;DR
This paper introduces a framework for "adaptable watermarking" that allows a base neural watermarking model to be transformed into a target model using lightweight, parameter-efficient adapters. By fine-tuning only ~1-2M parameters per component, the authors achieve an 8x reduction in model storage and loading times while maintaining near-oracle decoding accuracy and robustness across diverse watermarking styles.
What Problem Does This Paper Address?
Deep learning has revolutionized invisible image watermarking, but it has also led to a fragmented landscape of monolithic, incompatible models. Currently, if a platform wants to upgrade its watermarking scheme, interoperate with a third party, or offer different "profiles" (e.g., high fidelity vs. high robustness), it must retrain and deploy entirely new encoder-decoder pairs.
This is computationally expensive and creates significant infrastructure overhead. Shipping multiple large models (often ~100MB+ each) to clients is impractical. The authors address this "monolithic fragmentation" by asking: Can we transform one watermarking method into another without retraining the full model?
Methods
The core of the proposed solution is a modular framework that uses Convolutional-Weight LoRA (CWLoRA) adapters.
1. CWLoRA Adapters
While standard LoRA is typically applied to linear layers in Transformers, the authors adapt it for the convolutional backbones common in watermarking. CWLoRA applies low-rank updates directly to convolutional kernels. Crucially, these updates can be merged into the base weights at inference time, introducing zero additional latency.
2. Multi-Domain Adaptation Loss
To make a base encoder ($E_{base}$) mimic a target encoder ($E_{target}$), the authors use a novel multi-domain residual alignment loss ($\mathcal{L}_{rec}$). This loss compares the "watermark residuals" (the difference between the watermarked and original image) across four domains:
- Spatial: Pixel-wise similarity.
- Gradient: Matching edges and textures using Sobel operators.
- Frequency: Aligning patterns in the Fourier domain.
- Cosine Similarity: Ensuring the direction of the residual vectors matches.
Simultaneously, a Secret Loss ($\mathcal{L}_{sec}$) ensures that the adapted decoder can still accurately recover the embedded bit messages, even under simulated distortions (noise).
Datasets
- MIRFLICKR: Used for primary training (80k images) and testing (10k images).
- Defactify Image Dataset: Used for cross-dataset evaluation to ensure the adapters learn general watermark transformations rather than overfitting to specific image distributions.
- Target Models: The framework was evaluated by adapting between several state-of-the-art models: TrustMark (P/Q), InvisMark, RoSteALS, HiDDeN, and WAM.
Key Insights and Ideas
- Behavioral Mimicry vs. Parameter Matching: The authors show that you don't need to match the internal parameters of a target model to replicate its behavior. By aligning the output residuals, a lightweight adapter can effectively "re-skin" a base model.
- The Necessity of Priors: One of the most interesting findings is that a generic ImageNet-pretrained ResNet cannot be easily adapted to highly imperceptible watermarks like InvisMark. The base model needs "watermarking priors"—pre-existing knowledge of how to hide data in high-frequency artifacts—to be successfully adapted.
- Interoperability as a Feature: Adapted models are naturally interoperable. An adapted encoder's output can be read by the original target decoder, and an adapted decoder can read messages from the original target encoder.
- Efficiency Beyond Parameter Count: CWLoRA outperformed other common adapters (like IA3 or 1x1 bottlenecks). The authors suggest that direct weight-space updates are better suited for the delicate, high-frequency nature of watermarking signals.
Key Results
- Massive Footprint Reduction: In a multi-decoder scenario (supporting $N$ different watermark styles), the system footprint was reduced from ~1.2 GB to ~0.14 GB—an 8x improvement.
- Near-Oracle Accuracy: Adapted decoders achieved bit accuracies within 0.6–1.0% of the original monolithic target models.
- Robustness Transfer: Adapted models successfully inherited the robustness profiles of their targets, maintaining high accuracy even under heavy JPEG compression, blurring, and geometric distortions.
- Parameter Efficiency: High-quality adaptation was achieved with as few as 0.9M (encoder) and 2.0M (decoder) trainable parameters, roughly 10% of a typical base model's size.
Limitations and Future Work
- Black-Box Adaptation: Currently, the method requires white-box access to the target model to calculate the residual alignment loss. Future work could explore adapting to targets available only via API or binary.
- Architecture Specificity: While CWLoRA works great for CNNs, the paper notes that "native ViT adapters" would be needed for transformer-based watermarking backbones.
- Generative Models: The current study focuses on post-generation watermarking. Extending this to "in-generation" watermarks (e.g., for Stable Diffusion) is a promising next step.
Summary updated