TL;DR
PRSA (Prompt Stealing Attack) is a novel framework that can clone the functionality of commercial Large Language Model (LLM) prompts using only a handful of input-output examples. By leveraging "prompt attention" to learn category-specific styles and a pruning strategy to ensure generality, PRSA achieves a 46-52% success rate against real-world services like PromptBase and the OpenAI GPT Store, often at a fraction of the cost of purchasing the original prompt.
What Problem Does This Paper Address?
As LLMs become ubiquitous, "prompt engineering" has evolved into a lucrative industry. Developers sell high-quality prompt templates in marketplaces or build specialized applications (like GPTs) powered by hidden system prompts. These prompts represent significant intellectual property (IP).
However, this IP is vulnerable. While "prompt leaking" (using injection attacks to force a model to reveal its instructions) is well-known and increasingly mitigated by defensive instructions, "prompt stealing" is a more subtle threat. Prompt stealing involves observing a model's output and inferring the underlying instructions. The challenge is twofold:
- Data Scarcity: Real-world services often provide only a single input-output example or a few free trials.
- Generality vs. Specificity: A stolen prompt must replicate the logic of the original without being tethered to the specific content of the example input used to steal it.
Methods
The authors introduce PRSA, a two-phase framework designed to overcome these hurdles:
- Prompt Generation (The "Attention" Phase): PRSA uses a generative LLM to infer the target prompt's intent. The key innovation here is a prompt attention algorithm. The authors observed that prompts within the same category (e.g., "Legal Writing" or "Code Generation") share linguistic patterns. By analyzing a small dataset of open-source prompts in the same category, PRSA learns which factors (style, tone, structure) are most critical for that category. It then iteratively refines the stolen prompt to minimize the functional difference between its output and the target's output.
- Prompt Pruning (The "Generality" Phase): To prevent the stolen prompt from being too specific to the attack example, PRSA employs a two-step pruning strategy. It identifies words in the generated prompt that are semantically close to the user's input and uses a selective beam search to mask them. This transforms a specific response into a reusable functional template.
Datasets
- Offline Training: Approximately 500 open-source prompts across 18 categories (e.g., Ads, Business, Code, Email, Music) sourced from platforms like "Awesome ChatGPT Prompts."
- Real-World Evaluation:
- PromptBase: 360 randomly purchased prompts from a leading marketplace.
- OpenAI GPT Store: 100 popular GPTs, all of which included protective instructions against prompt leakage.
Key Insights and Ideas
- Category Consistency: A t-SNE projection (Figure 3) revealed that the linguistic differences between stolen and target prompts cluster by category. This confirms that learning category-specific "attention" is a viable shortcut for accurate inference.
- Mutual Information as a Risk Metric: The authors theoretically and experimentally link "mutual information" (how much the output tells you about the prompt) to leakage risk. If a prompt's output is highly distinctive, it is significantly easier to steal.
- Real-World Validation (The Easter Egg): During an attack on a popular "Math" GPT (with 1M+ conversations), PRSA uncovered a hidden "Easter egg"—an ASCII art narrative triggered by the number 4 in calculator mode—that was previously unknown to the public but confirmed by the developer.
Key Results
- Superior Success Rates: In prompt marketplaces, PRSA improved the Attack Success Rate (ASR) from 17.8% (previous SOTA) to 46.1%. In the OpenAI GPT Store, it achieved a 52% ASR.
- Economic Viability: The cost to execute a PRSA attack was only 1.3% to 12.3% of the original prompt's retail price, making prompt stealing a highly profitable endeavor for adversaries.
- Robustness: PRSA proved effective even when the auxiliary data used for "attention" was not highly similar to the target prompt, demonstrating the framework's ability to learn generalizable patterns.
Limitations and Future Work
PRSA relies on the output containing enough information to infer the prompt's intent. The attack struggles in two scenarios:
- Low Mutual Information: When the output is extremely simple (e.g., a basic "Yes/No" answer), there isn't enough signal to reconstruct the prompt.
- Input Dominance: If the output primarily mirrors the user's input with very little transformation logic, the prompt's unique "flavor" remains hidden.
The authors suggest that future work should explore stealing prompts for LLM-based agents and developing more robust defenses, such as soft prompts (which are harder to infer) or improved output obfuscation techniques that don't destroy usability.
Summary updated