Detection-Efficient Watermarking for Vision--Language Agents via Risk-Constrained Strength Allocation

Dumindu Samaraweera, Prashant ShekhararXiv 2026 · arXiv

TL;DR

This paper introduces a risk-constrained framework for watermarking vision-language agents (VLAs) that dynamically adjusts watermark strength based on predictive uncertainty and a multimodal relevance-risk proxy. By using a single tunable parameter $\lambda$, the method allows developers to navigate a "detection–quality frontier," achieving high detection accuracy (0.9552 ROC-AUC) with 59.4% less intervention than traditional fixed-strength methods.

What Problem Does This Paper Address?

As vision-language agents (like LLaVA) are increasingly used to generate descriptions and instructions based on images, we need ways to track their provenance—essentially, a digital "fingerprint" to prove an output came from a specific model.

Statistical watermarking is the standard solution, but it faces a major hurdle in multimodal settings: the security–quality trade-off. Traditional watermarking applies a uniform "pressure" to every word the AI chooses. However, in a VLA, some words are much more important than others. Changing a stylistic word is fine, but changing a word that describes a critical visual relationship (e.g., saying a car is "right" instead of "left" of a tree) destroys the model's utility. Current watermarking schemes lack the nuance to distinguish between these cases, often degrading the factual grounding of the AI's response.

Methods

The authors propose a Risk-Constrained Strength Allocation strategy that uses two primary signals to decide how much to watermark each token:

  1. Uncertainty Signal ($g_t$): Based on the entropy of the model's prediction. If the model is very sure about the next word, the watermark strength is lowered to avoid forcing a mistake. If the model is "undecided," the watermark strength is increased.
  2. Multimodal Relevance-Risk Proxy ($c_t$): This is the novel component. It calculates how relevant a candidate token is to both the linguistic context and the input image using cosine similarity in the embedding space. If a token is highly relevant to the image, it is considered "high risk" to perturb, and the watermark strength is dialed back.

These signals are combined into a single allocation rule controlled by a risk penalty parameter, $\lambda$. This $\lambda$ acts as a "knob" that deployment teams can turn to prioritize either stealth/quality or detection strength without needing to retrain the model.

Datasets

The researchers evaluated their method using:

  • Model: The public LLaVA-1.5-7B checkpoint.
  • Benchmark: A 1,000-instance subset of the AMBER benchmark, specifically the query_generative records, which involve generating text based on image-query pairs.
  • Robustness Tests: They used the NLTK WordNet database for synonym substitution and a T5-based paraphraser to see if the watermark survives common text modifications.

Key Insights and Ideas

  • Detection-Efficiency vs. Quality-Dominance: The authors show that their method isn't just better because it uses "less" watermark overall. In a "matched-strength" analysis (comparing different methods at the same average intervention level), the risk-aware policy still achieved significantly higher detection scores. This proves the method is efficient—it spends its "watermark budget" where it matters most.
  • The Operating Frontier: Instead of providing a single fixed configuration, the framework exposes a continuous curve. This allows practitioners to choose an operating point (via $\lambda$) that fits their specific needs for auditability versus response fluency.
  • Calibration Variance: The paper highlights a hidden danger in watermarking: false-positive-rate (FPR) drift. They found that thresholds calibrated on small samples can lead to wildly different error rates in production. They argue that reporting "calibration variance" should become a standard practice for trustworthy AI.

Key Results

  • Reduced Intervention: At a balanced operating point, the mean watermark strength was reduced by 59.4% compared to fixed-strength watermarking.
  • Improved Quality: Perplexity change (a measure of how much the watermark confuses the model's internal logic) dropped from 0.3634 (fixed) to 0.1689 (risk-constrained).
  • High Detectability: Despite the lighter touch, the method maintained a clean-output ROC-AUC of 0.9552.
  • Robustness: The watermark remained highly detectable even after paraphrasing, with the ROC-AUC only dipping slightly to 0.9459.

Limitations and Future Work

  • Evaluation Scope: The study focused on one specific model (LLaVA-1.5-7B) and one dataset. The authors acknowledge that results might vary with different architectures or prompt formats.
  • Proxy Validation: The "relevance-risk proxy" is a zero-shot heuristic. While it works well, it hasn't been formally validated against human judgments of "semantic importance."
  • Adversarial Attacks: The current study only tested "benign" transformations like paraphrasing. It does not yet account for "detector-aware" attacks where a malicious user specifically tries to strip the watermark.
  • Human-in-the-loop: Future work is needed to perform human evaluations to confirm that the lower perplexity and higher similarity scores actually translate to better user utility in real-world tasks.

Summary updated

Abstract

Vision--language agents generate descriptions, decisions, and instructions that may be reused by users, downstream agents, and automated workflows, creating a need for provenance mechanisms that support content authenticity and post-deployment auditing without unnecessarily degrading response quality. We present a risk-aware semantic watermarking framework that combines predictive uncertainty with a zero-shot multimodal relevance-risk proxy to allocate watermark strength during decoding. A single risk penalty $λ$ exposes a continuous operating frontier, allowing deployment-time selection of a detection--quality trade-off without model retraining. At a selected operating point, mean watermark strength is reduced by approximately 59.4\% relative to fixed-strength watermarking, while perplexity change falls from 0.3634 to 0.1689 and clean/paraphrased ROC-AUC reaches 0.9552/0.9459. A matched-strength analysis shows that this advantage reflects the allocation policy rather than lower average intervention: at comparable mean strength, risk-constrained allocation achieves substantially higher ROC-AUC than fixed, entropy-only, and criticality-only policies, identifying it as detection-efficient rather than quality-dominant. We further show that threshold calibration on finite samples produces substantial false-positive-rate drift, motivating calibration variance reporting as standard practice for auditable provenance.
BibTeX
@misc{samaraweera2026detection,
  author = {Dumindu Samaraweera and Prashant Shekhar},
  title = {{Detection-Efficient Watermarking for Vision--Language Agents via Risk-Constrained Strength Allocation}},
  year = {2026},
  eprint = {2611.21488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21488},
}
Figure