PRSA: Prompt Stealing Attacks against Real-World Prompt Services

Yong Yang, Changjiang Li, Qingming Li, Oubo Ma, Haoyu Wang, Zonghui Wang, Yandong Gao, Wenzhi Chen, Shouling Ji

TL;DR

PRSA (Prompt Stealing Attack) is a novel framework that can clone the functionality of commercial Large Language Model (LLM) prompts using only a handful of input-output examples. By leveraging "prompt attention" to learn category-specific styles and a pruning strategy to ensure generality, PRSA achieves a 46-52% success rate against real-world services like PromptBase and the OpenAI GPT Store, often at a fraction of the cost of purchasing the original prompt.

What Problem Does This Paper Address?

As LLMs become ubiquitous, "prompt engineering" has evolved into a lucrative industry. Developers sell high-quality prompt templates in marketplaces or build specialized applications (like GPTs) powered by hidden system prompts. These prompts represent significant intellectual property (IP).

However, this IP is vulnerable. While "prompt leaking" (using injection attacks to force a model to reveal its instructions) is well-known and increasingly mitigated by defensive instructions, "prompt stealing" is a more subtle threat. Prompt stealing involves observing a model's output and inferring the underlying instructions. The challenge is twofold:

  1. Data Scarcity: Real-world services often provide only a single input-output example or a few free trials.
  2. Generality vs. Specificity: A stolen prompt must replicate the logic of the original without being tethered to the specific content of the example input used to steal it.
Figure 2: Comparison of prompt leaking attacks and prompt stealing attacks in unauthorized access to system prompts.

Methods

The authors introduce PRSA, a two-phase framework designed to overcome these hurdles:

  1. Prompt Generation (The "Attention" Phase): PRSA uses a generative LLM to infer the target prompt's intent. The key innovation here is a prompt attention algorithm. The authors observed that prompts within the same category (e.g., "Legal Writing" or "Code Generation") share linguistic patterns. By analyzing a small dataset of open-source prompts in the same category, PRSA learns which factors (style, tone, structure) are most critical for that category. It then iteratively refines the stolen prompt to minimize the functional difference between its output and the target's output.
  2. Prompt Pruning (The "Generality" Phase): To prevent the stolen prompt from being too specific to the attack example, PRSA employs a two-step pruning strategy. It identifies words in the generated prompt that are semantically close to the user's input and uses a selective beam search to mask them. This transforms a specific response into a reusable functional template.
Figure 4: Overview of PRSA.

Datasets

  • Offline Training: Approximately 500 open-source prompts across 18 categories (e.g., Ads, Business, Code, Email, Music) sourced from platforms like "Awesome ChatGPT Prompts."
  • Real-World Evaluation:
    • PromptBase: 360 randomly purchased prompts from a leading marketplace.
  • OpenAI GPT Store: 100 popular GPTs, all of which included protective instructions against prompt leakage.

Key Insights and Ideas

  • Category Consistency: A t-SNE projection (Figure 3) revealed that the linguistic differences between stolen and target prompts cluster by category. This confirms that learning category-specific "attention" is a viable shortcut for accurate inference.
  • Mutual Information as a Risk Metric: The authors theoretically and experimentally link "mutual information" (how much the output tells you about the prompt) to leakage risk. If a prompt's output is highly distinctive, it is significantly easier to steal.
  • Real-World Validation (The Easter Egg): During an attack on a popular "Math" GPT (with 1M+ conversations), PRSA uncovered a hidden "Easter egg"—an ASCII art narrative triggered by the number 4 in calculator mode—that was previously unknown to the public but confirmed by the developer.
Figure 3: t-SNE projection of the differences between outputs from stolen and target prompts. The stolen prompts are gener- ated by GPT-3.5.

Key Results

  • Superior Success Rates: In prompt marketplaces, PRSA improved the Attack Success Rate (ASR) from 17.8% (previous SOTA) to 46.1%. In the OpenAI GPT Store, it achieved a 52% ASR.
  • Economic Viability: The cost to execute a PRSA attack was only 1.3% to 12.3% of the original prompt's retail price, making prompt stealing a highly profitable endeavor for adversaries.
  • Robustness: PRSA proved effective even when the auxiliary data used for "attention" was not highly similar to the target prompt, demonstrating the framework's ability to learn generalizable patterns.

Limitations and Future Work

PRSA relies on the output containing enough information to infer the prompt's intent. The attack struggles in two scenarios:

  1. Low Mutual Information: When the output is extremely simple (e.g., a basic "Yes/No" answer), there isn't enough signal to reconstruct the prompt.
  2. Input Dominance: If the output primarily mirrors the user's input with very little transformation logic, the prompt's unique "flavor" remains hidden.

The authors suggest that future work should explore stealing prompts for LLM-based agents and developing more robust defenses, such as soft prompts (which are harder to infer) or improved output obfuscation techniques that don't destroy usability.

Summary updated

Abstract

Recently, large language models (LLMs) have garnered widespread attention for their exceptional capabilities. Prompts are central to the functionality and performance of LLMs, making them highly valuable assets. The increasing reliance on high-quality prompts has driven significant growth in prompt services. However, this growth also expands the potential for prompt leakage, increasing the risk that attackers could replicate original functionalities, create competing products, and severely infringe on developers' intellectual property. Despite these risks, prompt leakage in real-world prompt services remains underexplored. In this paper, we present PRSA, a practical attack framework designed for prompt stealing. PRSA infers the detailed intent of prompts through very limited input-output analysis and can successfully generate stolen prompts that replicate the original functionality. Extensive evaluations demonstrate PRSA's effectiveness across two main types of real-world prompt services. Specifically, compared to previous works, it improves the attack success rate from 17.8% to 46.1% in prompt marketplaces and from 39% to 52% in LLM application stores, respectively. Notably, in the attack on "Math", one of the most popular educational applications in OpenAI's GPT Store with over 1 million conversations, PRSA uncovered a hidden Easter egg that had not been revealed previously. Besides, our analysis reveals that higher mutual information between a prompt and its output correlates with an increased risk of leakage. This insight guides the design and evaluation of two potential defenses against the security threats posed by PRSA. We have reported these findings to the prompt service vendors, including PromptBase and OpenAI, and actively collaborate with them to implement defensive measures.
BibTeX
@inproceedings{yang2025prsa,
  author = {Yong Yang and Changjiang Li and Qingming Li and Oubo Ma and Haoyu Wang and Zonghui Wang and Yandong Gao and Wenzhi Chen and Shouling Ji},
  title = {{PRSA: Prompt Stealing Attacks against Real-World Prompt Services}},
  booktitle = {USENIX Security},
  year = {2025},
  url = {https://openreview.net/forum?id=4QoNChHivA},
}
Figure