Adaptable Watermarking: Transforming Image Watermarking Models via Lightweight Adapters

Mehrdad Saberi, Vishal Asnani, Shruti Agarwal, Soheil Feizi, John CollomossearXiv 2026 · arXiv

TL;DR

This paper introduces a framework for "adaptable watermarking" that allows a base neural watermarking model to be transformed into a target model using lightweight, parameter-efficient adapters. By fine-tuning only ~1-2M parameters per component, the authors achieve an 8x reduction in model storage and loading times while maintaining near-oracle decoding accuracy and robustness across diverse watermarking styles.

What Problem Does This Paper Address?

Deep learning has revolutionized invisible image watermarking, but it has also led to a fragmented landscape of monolithic, incompatible models. Currently, if a platform wants to upgrade its watermarking scheme, interoperate with a third party, or offer different "profiles" (e.g., high fidelity vs. high robustness), it must retrain and deploy entirely new encoder-decoder pairs.

This is computationally expensive and creates significant infrastructure overhead. Shipping multiple large models (often ~100MB+ each) to clients is impractical. The authors address this "monolithic fragmentation" by asking: Can we transform one watermarking method into another without retraining the full model?

Methods

The core of the proposed solution is a modular framework that uses Convolutional-Weight LoRA (CWLoRA) adapters.

1. CWLoRA Adapters

While standard LoRA is typically applied to linear layers in Transformers, the authors adapt it for the convolutional backbones common in watermarking. CWLoRA applies low-rank updates directly to convolutional kernels. Crucially, these updates can be merged into the base weights at inference time, introducing zero additional latency.

2. Multi-Domain Adaptation Loss

To make a base encoder ($E_{base}$) mimic a target encoder ($E_{target}$), the authors use a novel multi-domain residual alignment loss ($\mathcal{L}_{rec}$). This loss compares the "watermark residuals" (the difference between the watermarked and original image) across four domains:

  • Spatial: Pixel-wise similarity.
  • Gradient: Matching edges and textures using Sobel operators.
  • Frequency: Aligning patterns in the Fourier domain.
  • Cosine Similarity: Ensuring the direction of the residual vectors matches.

Simultaneously, a Secret Loss ($\mathcal{L}_{sec}$) ensures that the adapted decoder can still accurately recover the embedded bit messages, even under simulated distortions (noise).

Method overview showing the training of adapters to mimic target residuals and the multi-decoder inference pipeline.

Datasets

  • MIRFLICKR: Used for primary training (80k images) and testing (10k images).
  • Defactify Image Dataset: Used for cross-dataset evaluation to ensure the adapters learn general watermark transformations rather than overfitting to specific image distributions.
  • Target Models: The framework was evaluated by adapting between several state-of-the-art models: TrustMark (P/Q), InvisMark, RoSteALS, HiDDeN, and WAM.

Key Insights and Ideas

  • Behavioral Mimicry vs. Parameter Matching: The authors show that you don't need to match the internal parameters of a target model to replicate its behavior. By aligning the output residuals, a lightweight adapter can effectively "re-skin" a base model.
  • The Necessity of Priors: One of the most interesting findings is that a generic ImageNet-pretrained ResNet cannot be easily adapted to highly imperceptible watermarks like InvisMark. The base model needs "watermarking priors"—pre-existing knowledge of how to hide data in high-frequency artifacts—to be successfully adapted.
  • Interoperability as a Feature: Adapted models are naturally interoperable. An adapted encoder's output can be read by the original target decoder, and an adapted decoder can read messages from the original target encoder.
  • Efficiency Beyond Parameter Count: CWLoRA outperformed other common adapters (like IA3 or 1x1 bottlenecks). The authors suggest that direct weight-space updates are better suited for the delicate, high-frequency nature of watermarking signals.

Key Results

  • Massive Footprint Reduction: In a multi-decoder scenario (supporting $N$ different watermark styles), the system footprint was reduced from ~1.2 GB to ~0.14 GB—an 8x improvement.
  • Near-Oracle Accuracy: Adapted decoders achieved bit accuracies within 0.6–1.0% of the original monolithic target models.
  • Robustness Transfer: Adapted models successfully inherited the robustness profiles of their targets, maintaining high accuracy even under heavy JPEG compression, blurring, and geometric distortions.
  • Parameter Efficiency: High-quality adaptation was achieved with as few as 0.9M (encoder) and 2.0M (decoder) trainable parameters, roughly 10% of a typical base model's size.
Figure 2. Adapter comparison for decoder-only adaptation. Bit accuracy under noise vs. number of parameters for decoder adaptation with different types of adapters. The base decoder is from the TrustMarkQ method.

Limitations and Future Work

  • Black-Box Adaptation: Currently, the method requires white-box access to the target model to calculate the residual alignment loss. Future work could explore adapting to targets available only via API or binary.
  • Architecture Specificity: While CWLoRA works great for CNNs, the paper notes that "native ViT adapters" would be needed for transformer-based watermarking backbones.
  • Generative Models: The current study focuses on post-generation watermarking. Extending this to "in-generation" watermarks (e.g., for Stable Diffusion) is a promising next step.

Summary updated

Abstract

We explore the adaptability of neural image watermarking models; the extent to which one watermarking method can be transformed into another through lightweight adaptation. We introduce a framework that enables encoder-decoder watermarking architectures to become modular and upgradable, supporting multiple watermarking styles without re-training full models. Adaptation is performed by attaching a compact parameter-efficient adapter and fine-tuning under a multi-domain residual alignment loss (spatial, gradient, and frequency) while preserving bit accuracy via a secret-loss term. We evaluate this approach across a suite of recent watermarking models, including TrustMark, InvisMark, RoSteALS, HiDDeN, and WAM, demonstrating that adapted encoders closely reproduce target residuals and that adapted decoders recover messages with near-optimal accuracy under standard augmentations and distortions. We show that cross-model adaptation, with applications in interoperable and upgradable watermarking, can be achieved with as few as 1-2M trainable parameters per component (e.g., the 2M decoder adapter used in our multi-decoder inference pipeline).
BibTeX
@misc{saberi2026adaptable,
  author = {Mehrdad Saberi and Vishal Asnani and Shruti Agarwal and Soheil Feizi and John Collomosse},
  title = {{Adaptable Watermarking: Transforming Image Watermarking Models via Lightweight Adapters}},
  year = {2026},
  eprint = {2611.18363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18363},
}
Figure