SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

Yifei Zhao, Qian Lou, Mengxin Zheng

TL;DR

Existing fingerprinting methods for protecting Large Vision-Language Models (LVLMs) from intellectual property theft are easily bypassed because they rely on unnatural, out-of-distribution triggers. This paper introduces SIF (Semantically In-Distribution Fingerprints), a non-intrusive framework that embeds hidden text watermarks directly into natural-looking images. This approach makes the fingerprints highly robust to downstream model modifications like fine-tuning and virtually invisible to malicious actors.

What Problem Does This Paper Address?

Training state-of-the-art Large Vision-Language Models (LVLMs) requires massive datasets and immense computational power. To foster open research, organizations frequently release these models under specific licenses. However, malicious actors can easily violate these agreements, deploying open-source models to build unauthorized commercial APIs.

To detect this, developers use model fingerprinting—generating specific input-output pairs to verify ownership. Unfortunately, existing methods rely on semantically abnormal queries (like gibberish prompt prefixes) or force the model to output text completely unrelated to the image (e.g., looking at a picture of a skier and outputting "CVPR Conference").

The authors expose this flaw by introducing the Semantic Divergence Attack (SDA). They demonstrate that an attacker can easily use a lightweight reference model to flag and filter out these unnatural fingerprinting attempts by measuring perplexity and semantic divergence, effectively neutralizing current IP protection schemes.

Comparison of existing fingerprinting methods vs. the proposed SIF framework

Methods

SIF circumvents the detection problem by keeping both the query and the response entirely within the model's natural semantic distribution. It operates without modifying the underlying model parameters through two core components:

  1. Semantic-Aligned Fingerprint Distillation (SAFD): Instead of altering text prompts, SAFD applies an imperceptible pixel perturbation to the input image. This perturbation distills a standard text-decoding watermark (which subtly biases token selection toward a secret "green list" of words) into the visual modality. The model generates a completely natural, context-aware description of the image, but the text secretly carries a statistically verifiable watermark.
  2. Robust-Fingerprint Optimization (RFO): Stolen models are often quantized or fine-tuned, which alters intermediate activations and can erase standard watermarks. RFO combats this by simulating a "worst-case" representation perturbation during the image optimization phase. By anticipating internal embedding drift, it ensures the visual trigger remains effective even after the model undergoes heavy modification.
Figure 3. The overview of our proposed method. (a) Fingerprint construction. We generate fingerprint queries by applying Semantic- Aligned Fingerprint Distillation (SAFD) and Robust-Fingerprint Optimization (RFO), which distill decoding-based text watermarks into input images while resisting worst-case embedding perturbations. (b) Copyright verification. The trigger image is queried to a suspected stolen LVLM. If the generated text contains the embedded watermark signal under standard decoding, model ownership is verified.

Datasets

The authors evaluated SIF using the following datasets:

  • Trigger Dataset: Initialized using regular images from the ImageNet 2012 validation set paired with generation-task prompts from the AMBER benchmark to create 200 unique fingerprint queries.
  • Downstream Fine-Tuning Datasets: To evaluate robustness against malicious user modifications, they tested models fine-tuned on LlavaMix (multimodal instruction-following), TikZ (diagram-to-code), GUI-Actor (GUI grounding), ARC-AGI-1 (abstract reasoning), V7W (grounded VQA), TextVQA, Paintingform, and MathV (mathematical reasoning).

Key Insights and Ideas

  • The Visual Modality as a Stealth Carrier: Instead of hacking the text decoding process—which is impossible if you only have black-box API access to a suspect model—you can bake the watermarking instructions directly into the image pixels.
  • Stealthiness via Top-K Constraints: By truncating token optimization to the natural top-$K$ decoding region, SIF prevents the model from drifting into irrelevant semantics, ensuring the output looks exactly like a normal human-model interaction.
  • Adversarial Robustness for Embeddings: RFO borrows the philosophy of adversarial training but applies it internally. By optimizing the image against the most harmful potential embedding shifts, the fingerprint survives downstream model compression.

Key Results

The authors evaluated SIF against popular models like LLaVA-1.5-7B and Qwen2.5-VL-7B, yielding several key findings:

  • Defeating the Semantic Divergence Attack: Under the proposed SDA defense, prior fingerprinting methods (like Proflingo and PLA) dropped to a 0% Fingerprint Matching Rate (FMR). SIF maintained a resilient FMR of 85% to 86%.
  • Evading Advanced AI Judges: When using a GPT-4-based semantic judge to flag suspicious interactions, 86% of PLA queries were caught, whereas only 4% of SIF interactions were flagged.
  • Resilience to Quantization and Fine-Tuning: On LLaVA-1.5, SIF achieved an FMR of 0.49 under 4-bit quantization and 0.89 under 8-bit quantization. Crucially, while baseline methods dropped to 0.00 FMR after full fine-tuning on LlavaMix, SIF retained its tracking capability.
Figure 4. Example: Given the same fingerprint query, the unrelated model produces outputs without a detectable watermark signal, whereas suspect models generate semantically natural outputs that nevertheless exhibit significant fingerprint signals.

Limitations and Future Work

  • Not a Complete Legal Proof: The authors explicitly note that model fingerprinting should be viewed as a practical first-step screening tool. Definitive legal verification still requires hard evidence like training logs, compute bills, or dataset records.
  • Modality Constraints: SIF is currently tailored for vision-to-text models (LVLMs). The authors note that extending this technique to video and image generation models represents an open challenge, primarily due to the complexities of maintaining temporal consistency.

Summary updated

Abstract

The public accessibility of large vision-language models (LVLMs) raises serious concerns about unauthorized model reuse and intellectual property infringement. Existing ownership verification methods often rely on semantically abnormal queries or out-of-distribution responses as fingerprints, which can be easily detected and removed by adversaries. We expose this vulnerability through a Semantic Divergence Attack (SDA), which identifies and filters fingerprint queries by measuring semantic divergence between a suspect model and a reference model, showing that existing fingerprints are not semantic-preserving and are therefore easy to detect and bypass. To address these limitations, we propose SIF (Semantically In-Distribution Fingerprints), a non-intrusive ownership verification framework that requires no parameter modification. SIF introduces Semantic-Aligned Fingerprint Distillation (SAFD), which transfers text watermarking signals into the visual modality to produce semantically coherent yet fingerprinted responses. In addition, Robust-Fingerprint Optimization (RFO) enhances robustness by simulating worst-case representation perturbations, making the fingerprints resilient to model modifications such as fine-tuning and quantization. Extensive experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate that SIF achieves strong stealthiness and robustness, providing a practical solution for LVLM copyright protection. Code is available at https://github.com/UCF-ML-Research/SIF-VLM-Fingerprint
BibTeX
@inproceedings{zhao2026sif,
  author = {Yifei Zhao and Qian Lou and Mengxin Zheng},
  title = {{SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=6n0JioA7Gd},
}
Figure