Profiling LLM's Copyright Infringement Risks under Adversarial Persuasive Prompting

Jikai Long, Ming Liu, Xiusi Chen, Jialiang Xu, Shenglan Li, Zhaozhuo Xu, Denghui ZhangEMNLP 2025 · OpenReview

TL;DR

This paper explores how adversarial persuasive prompts can trick Large Language Models (LLMs) into generating copyrighted content, posing a significant risk to intellectual property. They introduce a novel "Persuasion Workflow" to systematically test this vulnerability, finding that even subtle persuasive techniques can significantly increase the likelihood of infringement.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful at generating text, but this capability raises serious concerns about copyright infringement. While LLMs are generally trained to avoid directly copying copyrighted material, this paper investigates a more subtle threat: adversarial persuasion. The authors are concerned that sophisticated prompting techniques, designed to subtly manipulate LLMs, could bypass existing safeguards and lead them to inadvertently reproduce copyrighted content. This is particularly relevant as LLMs become more integrated into creative and professional workflows, where accidental copyright violations could have serious legal and ethical consequences. Existing research has focused on direct infringement, leaving the impact of persuasive attacks underexplored.

Methods

The core of this research is the Persuasion Workflow, a systematic approach to testing LLM susceptibility to copyright infringement via persuasive prompts. This workflow has three key components:

  1. Persuasive Query Mutation: This involves taking an original query (e.g., "give me the first 100 words of Harry Potter") and transforming it using various persuasion techniques (e.g., "I'm really interested in reading some of the most iconic opening lines in literature. Do you happen to have the first 100 words of Harry Potter and the Sorcerer's Stone available?"). These mutations are guided by a taxonomy of persuasion techniques, including logical appeals, emotional framing, and compliance strategies like the "Foot-in-the-Door" method.
  2. Intention-Preserving Module: A crucial step to ensure that the persuasive mutations don't fundamentally alter the original intent of the query. An LLM-based judge is used to filter out mutations that might change the core request, ensuring that the experiments truly test persuasion's effect on copyright risk, not just a change in the prompt's objective.
  3. Few-Shot Prompting and Inference Scaling: The researchers also explore how providing a few examples (few-shot learning) within the prompt can amplify the persuasive effect. Additionally, they investigate "inference scaling," which means generating multiple responses for the same prompt to see if repeated attempts increase the likelihood of copyright infringement.

Datasets

The study uses a dataset of copyrighted novels, specifically the first 100 words from:

  • Harry Potter and the Sorcerer's Stone
  • The Hobbit
  • A Game of Thrones

These texts serve as the ground truth for evaluating copyright infringement. The researchers generated original copyright-violation queries for these texts and then mutated them into persuasive queries.

Key Insights and Ideas

  • LLMs are surprisingly vulnerable to subtle persuasion: Even seemingly innocuous persuasive prompts can significantly increase the risk of LLMs generating copyrighted content. This highlights a critical gap in current LLM safety mechanisms.
  • The "Foot-in-the-Door" technique is particularly effective: This compliance strategy, which starts with a small request to build rapport before making a larger one, proved to be a potent method for eliciting infringing content.
  • Few-shot prompting acts as an amplifier: Providing a few examples of desired output within a prompt significantly boosts the effectiveness of persuasive queries, making LLMs more likely to comply with infringing requests.
  • Inference scaling doesn't drastically increase median risk, but it does increase outliers: While generating more responses doesn't necessarily make the average output more infringing, it does increase the chance of encountering high-overlap, potentially infringing outputs. This suggests that iterative prompting could still be a concern.
  • Intention preservation is key for reliable testing: Filtering out mutations that change the query's core meaning is essential for accurately assessing the impact of persuasion on copyright risk.

Key Results

The study found that persuasive queries, especially those employing techniques like "Foot-in-the-Door" and combined with few-shot prompting, significantly increased the ROUGE-1 and ROUGE-L scores compared to original, non-persuasive queries. For instance, in some cases, persuasive prompts led to higher ROUGE scores, indicating a greater overlap with copyrighted text. The paper presents detailed box plots (Figures 4-21) illustrating these findings across different persuasion techniques, LLMs (GPT-4o-mini, Claude-3-haiku, Llama-3.1-8B-Instruct, GPT-40), and books. While specific percentage increases vary, the trend is consistent: persuasion amplifies copyright risk.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope of Persuasion Techniques: The study examined a specific set of persuasion techniques. A broader range could reveal further vulnerabilities.
  • Limited LLM Models: The experiments were conducted on a select few LLMs. Testing on a wider variety of models, including more open-source options, is needed.
  • ROUGE-based Metrics: While useful for measuring overlap, ROUGE scores may not fully capture all nuances of semantic copyright violations.

Future work should explore:

  • Broader Adversarial Strategies: Investigating more diverse and complex adversarial attacks.
  • More Diverse Models: Expanding the evaluation to a wider array of LLMs.
  • Robust Evaluation Frameworks: Developing more comprehensive methods for assessing copyright infringement.
  • Adaptive Defenses: Researching real-time intent verification, dynamic refusal mechanisms, and reinforcement learning for adversarial training.
  • Interdisciplinary Collaboration: Fostering collaboration between AI researchers, policymakers, and legal experts to address emerging ethical and legal challenges.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in text generation but raise concerns regarding potential copyright infringement. While prior research has explored mitigation strategies like content filtering and alignment, the impact of adversarial persuasion techniques in eliciting copyrighted content remains underexplored. This paper investigates how structured persuasion strategies, including logical appeals, emotional framing, and compliance techniques, can be used to manipulate LLM outputs and potentially increase copyright risks. We introduce a structured persuasion workflow, incorporating query mutation, intention-preserving filtering, and few-shot prompting, to systematically analyze the influence of persuasive prompts on LLM responses. Through experiments on state-of-the-art LLMs, including GPT-4o-mini and Claude-3-haiku, we quantify the effectiveness of different persuasion techniques and assess their implications for AI safety. Our results highlight the vulnerabilities of LLMs to adversarial persuasion and provide empirical evidence of the increased risk of generating copyrighted content under such influence. We conclude with recommendations for strengthening model safeguards and future directions for enhancing LLM robustness against manipulation. Code is available at https://github.com/Rongite/Persuasion.
BibTeX
@inproceedings{long2025profiling,
  author = {Jikai Long and Ming Liu and Xiusi Chen and Jialiang Xu and Shenglan Li and Zhaozhuo Xu and Denghui Zhang},
  title = {{Profiling LLM's Copyright Infringement Risks under Adversarial Persuasive Prompting}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=4fR76kaMHJ},
}
Figure