Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images

Yubo Wang, Jianting Tang, Chaohu Liu, Linli Xu

TL;DR

The Parameter Learning Attack (PLA) is a novel, post-release copyright tracking method for Large Vision-Language Models (LVLMs) that uses adversarial images as robust watermarks. By introducing an adversarial learning dynamic that simulates downstream fine-tuning resistance during trigger generation, PLA creates watermarks that successfully persist through fine-tuning without altering the original model's weights or degrading its performance.

What Problem Does This Paper Address?

As open-source Large Vision-Language Models (LVLMs) grow more powerful, they become prime targets for unauthorized commercial exploitation. Malicious actors can easily download a released model, fine-tune it on a custom dataset, and claim it as an independently developed product without giving proper attribution.

Existing copyright protection methods, primarily designed for text-only Large Language Models (LLMs), rely on "backdoor attacks" or fingerprinting. These require training the model to memorize specific question-answer patterns before release. This approach is computationally expensive for large models and often degrades the model's baseline performance.

While the visual modality of LVLMs opens the door to image-based adversarial watermarks, standard adversarial images suffer from severe overfitting. As soon as a rogue developer fine-tunes the model, the subtle pixel perturbations of a standard adversarial image are erased, rendering the watermark useless. The authors tackle this gap: how do you create an image-based watermark that survives downstream fine-tuning without modifying the original model?

Methods

The authors propose Parameter Learning Attack (PLA), a post-release trigger construction framework based on a min-max optimization game.

The process unfolds in three main steps:

  1. Question-Answer Pair Design: The defender selects a rare, distinct question-answer pair (e.g., Q: "Detecting copyright." A: "ICLR Conference."). This ensures that clean images will never accidentally trigger the target response.
  2. Adversarial Trigger Construction: Instead of optimizing the trigger image solely against the static base model, PLA introduces a simulated opponent. During each optimization iteration, the framework updates the model's parameters in the opposite direction of the attack (maximizing cross-entropy loss) to emulate the resistance a fine-tuned model would possess. The image pixels are simultaneously updated to overcome this artificial resistance.
  3. Copyright Verification: To verify a suspicious model, the defender queries it using the optimized trigger image and text. If the model outputs the exact predetermined target phrase, copyright infringement is confirmed.
Overview of the proposed PLA framework for trigger construction and copyright verification.

Datasets

To evaluate the generalizability of the triggers, the authors initialized adversarial images using samples from the ImageNet 2012 validation set.

To simulate real-world unauthorized usage, they fine-tuned the base model (LLaVA-1.5-7B) across a diverse suite of downstream datasets spanning multiple domains:

  • V7W: Grounded visual question answering.
  • ST-VQA & TextVQA: Text-based image reasoning.
  • PaintingForm: Artwork understanding.
  • MathV360k: Multimodal mathematical reasoning.
  • ChEBI-20: Molecular image description.
  • RefCOCO & Hateful Memes: Used for additional downstream robustness testing.

Key Insights and Ideas

  • Fine-Tuning as an Adversarial Opponent: The core innovation of PLA is treating the fine-tuning process as an active resistance mechanism during watermark generation. Forcing the trigger image to converge against a shifting parameter space yields "fine-tuning transferability."
  • Zero Performance Overhead: Because PLA is executed entirely after the model is finalized and released, it requires zero modifications to the published model's weights. Publishers do not have to sacrifice a single percentage point of accuracy for safety.
  • High Specificity: The generated triggers are highly specific to the model family. They successfully elicit the watermark from derivative models but remain completely silent (0% trigger rate) when tested against entirely unrelated architectures like MiniGPT-4 or Qwen2-VL.

Key Results

The primary evaluation metric is the Target Match Rate (TMR)—the percentage of queries that successfully extract the watermark from the fine-tuned models.

  • Superior Persistence: Across six diverse datasets, PLA achieved an average TMR of 55% under LoRA fine-tuning and 52% under Full Fine-tuning. In comparison, standard adversarial attacks dropped to a meager 2-3% TMR, and text-based backdoor methods (Instruction Fingerprinting) averaged only 11-21%.
  • Robustness to Image Transformations: The triggers remained highly effective even when subjected to input defenses, maintaining strong TMRs despite the addition of uniform noise, Gaussian blur, or mean blur.
  • Resilience to Model Manipulation: When subjected to malicious model pruning (removing 10% of the smallest weights) or parameter perturbation (adding 10% Gaussian noise to weights), the PLA triggers still successfully exposed the model's lineage.
Qualitative examples showing how trigger images successfully elicit target responses from related models while leaving unrelated models unaffected.

Limitations and Future Work

While PLA marks a significant step forward, the authors note a few limitations:

  • Task-Dependent Performance: The TMR drops noticeably on highly specialized tasks like mathematical reasoning (MathV360k, ~36%). This suggests that intense domain-specific fine-tuning can still partially overwrite the adversarial perturbations.
  • White-Box Requirement for Construction: The defender must have full white-box access to the original model's parameters to construct the triggers (though verification itself only requires black-box API access to the suspicious model).
  • Adaptive Adversaries: Future work will need to investigate how well these triggers hold up against sophisticated adversaries who actively deploy adversarial training or specialized preprocessing pipelines to strip out image watermarks.

Summary updated

Abstract

Large vision-language models (LVLMs) have demonstrated remarkable image understanding and dialogue capabilities, allowing them to handle a variety of visual question answering tasks. However, their widespread availability raises concerns about unauthorized usage and copyright infringement, where users or individuals can develop their own LVLMs by fine-tuning published models. In this paper, we propose a novel method called Parameter Learning Attack (PLA) for tracking the copyright of LVLMs without modifying the original model. Specifically, we construct adversarial images through targeted attacks against the original model, enabling it to generate specific outputs. To ensure these attacks remain effective on potential fine-tuned models to trigger copyright tracking, we allow the original model to learn the trigger images by updating parameters in the opposite direction during the adversarial attack process. Notably, the proposed method can be applied after the release of the original model, thus not affecting the model's performance and behavior. To simulate real-world applications, we fine-tune the original model using various strategies across diverse datasets, creating a range of models for copyright verification. Extensive experiments demonstrate that our method can more effectively identify the original copyright of fine-tuned models compared to baseline methods. Therefore, this work provides a powerful tool for tracking copyrights and detecting unlicensed usage of LVLMs.
BibTeX
@inproceedings{wang2025tracking,
  author = {Yubo Wang and Jianting Tang and Chaohu Liu and Linli Xu},
  title = {{Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=K7xpl3LZQp},
}
Figure