Abstract
In recent years, "prompt as a service" has greatly enhanced the utility of large language models (LLMs) by enabling them to perform various downstream tasks efficiently without fine-tuning. This has also increased the commercial value of prompts. However, the potential risk of leakage in these commercialized prompts remains largely underexplored. In this paper, we introduce a novel attack framework, PRSA, designed for prompt stealing attacks against LLMs. The main idea of PRSA is to infer the intent behind a prompt by analyzing its input-output content, enabling the generation of a surrogate prompt that replicates the original's functionality. Specifically, PRSA mainly consists of two key phases: prompt mutation and prompt pruning. In the mutation phase, we propose a prompt attention algorithm based on output difference. The algorithm facilitates the generation of effective surrogate prompts by learning key factors that influence the accurate inference of prompt intent. During the pruning phase, we employ a two-step related word identification strategy to detect and mask words that are highly related to the input, thus improving the generalizability of the surrogate prompts. We verify the actual threat of PRSA through evaluation in both real-world settings, non-interactive and interactive prompt services. The results strongly confirm the PRSA's effectiveness and generalizability. We have reported these findings to prompt service providers and actively collaborate with them to implement defensive measures.
1. Introduction
The emergence of the "Prompt Economy" has turned well-crafted instructions into valuable intellectual property. Developers now monetize these prompts through marketplaces like PromptBase or specialized applications in the OpenAI GPT Store. While "prompt leaking"—using injection attacks to force a model to reveal its system instructions—is a recognized threat, developers have implemented increasingly robust defenses.
This paper introduces PRSA (Prompt Reverse Stealing Attack), a more sophisticated "black-box" threat. Unlike leaking, prompt stealing involves observing a model's output and reverse-engineering the underlying logic to create a functional surrogate.
2. Methodology
The PRSA framework operates through two primary phases designed to maximize the accuracy and generalizability of the stolen prompt.
2.1 Prompt Generation with "Prompt Attention"
PRSA leverages the concept of category consistency. By analyzing a dataset of open-source prompts across 18 categories (e.g., Ads, Code, Health), the system learns the "linguistic DNA" or attention factors—such as tone, style, and structure—relevant to specific domains. The prompt attention algorithm iteratively refines the stolen prompt by analyzing the semantic differences between the target model's output and the surrogate's output.
2.2 Prompt Pruning
A significant challenge in stealing prompts is "overfitting" to specific user inputs. For instance, a travel-related prompt might accidentally include specific location names from a test query. PRSA uses semantic similarity analysis and a selective beam search to identify and mask words that are too closely tied to the user's input, ensuring the final product is a versatile template rather than a one-off copy.
3. Experimental Results
The researchers evaluated PRSA using 360 prompts from PromptBase and 100 popular GPTs from the OpenAI Store.
- Attack Success Rate (ASR): PRSA achieved a 46.1% success rate in marketplaces and 52% in the GPT Store, significantly outperforming previous state-of-the-art methods (approx. 18% ASR).
- Economic Impact: The cost to replicate a prompt was found to be between 1.3% and 12.3% of its original purchase price. For a $4 prompt, the attack might cost only a few cents in API credits.
- Real-World Validation: The attack successfully uncovered a hidden "Easter egg" in a popular educational GPT with over 1 million conversations, confirming the framework's ability to extract deep logic.
4. Discussion and Conclusion
The study reveals a fundamental trade-off: high-utility prompts that produce complex, expressive outputs are inherently more vulnerable due to the high Mutual Information (MI) between the prompt and the output. Current defenses, such as output obfuscation, often degrade model utility to the point of making it unusable. Future research may need to explore "soft prompts"—mathematical vectors that are inherently resistant to human or LLM interpretation—to secure prompt-based intellectual property.
Summary updated