Reconstructing Hidden System Prompts from Language Model Outputs

29 concepts23 papers1 proposal

When interacting with a modern large language model (LLM) application, users rarely communicate with the raw model. Instead, user queries are typically wrapped inside a hidden "system prompt" that defines the application's persona, operational constraints, and safety guardrails. As Zixuan Xia et al. outline in The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective, these prompts have effectively become proprietary software programs.

However, because LLMs process system instructions and user data within the exact same context window, these hidden instructions are highly vulnerable to exposure. Researchers have uncovered two primary paradigms for extracting these secrets: Adversarial Prompt Extraction, which tricks the model into printing its own instructions, and Language Model Inversion, which mathematically reconstructs the prompt from the model's outputs.

The Mechanics of Adversarial Prompt Extraction

Why does a model leak its instructions when prompted deceptively? The core issue stems from instruction-data confusion. LLMs are optimized to be highly effective instruction followers, but they struggle to distinguish between the meta-instructions provided by the developer and the data inputs provided by the user.

As Junlin Wang et al. demonstrate in Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications, this creates a striking "capability-vulnerability paradox": the better a model is at following complex instructions, the easier it is to trick into revealing its own system prompt. Attackers exploit this by crafting inputs that override the original system instructions.

For example, Yu Peng et al. introduce RepeatLeakage: Leak Prompts from Repeating as Large Language Model Is a Good Repeater, which leverages the model's natural tendency to prioritize immediate repetition context over hidden constraints. By forcing the model into a repetitive loop, they can extract verbatim system prompts with near-perfect accuracy.

To scale and automate these vulnerabilities, researchers have moved beyond manual engineering:

Language Model Inversion and the Logit Leak

What if the attacker does not use adversarial text prompts at all? A deeper, more mathematical vulnerability exists: Language Model Inversion (LMI). Every time an LLM generates text, it computes a probability distribution over the entire vocabulary for the next token (known as logits).

In Language Model Inversion and its subsequent iteration Language Model Inversion, John Xavier Morris et al. (and John X. Morris et al.) reveal that these next-token probabilities act as a highly detailed "soft fingerprint" of the preceding hidden context. By training a separate encoder-decoder model (such as vec2text) to invert these probabilities, the authors show it is possible to reconstruct secret system prompts and private user inputs with high fidelity.

This vulnerability is magnified when APIs provide detailed logit distributions. As explored by Murtaza Nazir et al. in Better Language Model Inversion by Compactly Representing Next-Token Distributions, compact representations of these distributions make inversion attacks even more data-efficient.

To understand why this works geometrically, Haobo Wang et al. propose An Invariant Latent Space Perspective on Language Model Inversion. They treat the target LLM as a frozen, invariant decoder and train a lightweight inverse encoder within the model's own latent space, drastically reducing the data required to map outputs back to hidden inputs.

[Hidden System Prompt] + [User Input] 
                 │
                 ▼
       [Target Language Model]
                 │
                 ▼
    [Next-Token Probabilities (Logits)] ──┐
                 │                        │
                 ▼                        ▼
         [Inverse Encoder]        [Inverse Decoder]
                 │                        │
                 └───────────┬────────────┘
                             ▼
               [Reconstructed System Prompt]

The Semantic Footprint of Pure Text Outputs

A common defense against logit-based inversion is to restrict API access to pure text tokens, hiding the underlying probabilities. However, recent work shows that text alone is not a perfect shield.

Collin Zhang et al. introduce output2prompt in Extracting Prompts by Inverting LLM Outputs, demonstrating that hidden prompts leave distinct semantic footprints directly in the generated text. This allows zero-shot transfer attacks across completely different black-box models like GPT-4 and Llama-3 without needing internal logit data.

This phenomenon aligns with observations by Junbo Zhao et al. in DORY: Deliberative Prompt Recovery for LLM. The authors discover a fundamental correlation between low token uncertainty and the amount of "shared information" between the hidden prompt and the generated output. When a model generates highly confident text, it inherently leaks the structural constraints of the prompt that guided it.

This challenge is also mirrored in benign contexts. For example, Furkan Sahinuç et al. note in Systematic Task Exploration with LLMs: A Study in Citation Text Generation that providing detailed natural language intents inherently risks leaking elements of the desired output into the generation, highlighting how tightly bound instructions and outputs truly are.

Interestingly, this generative leakage behavior contrasts sharply with older, non-generative architectures. In A Study on Extracting Named Entities from Fine-tuned vs. Differentially Private Fine-tuned BERT Models, Andor Diera et al. find that encoder-only models like BERT are remarkably resistant to data extraction via generic prompts compared to autoregressive generative models, underscoring that prompt leakage is a vulnerability deeply tied to the mechanics of next-token generation.

Mitigating the Leakage

To counter these vulnerabilities, researchers are moving beyond simple text-based filtering. Zhixiong Zhuang et al. propose ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks, a defense mechanism that operates directly within the model's embedding space.

Instead of feeding the raw system prompt into the context window, they replace it with an optimized "proxy" embedding. This proxy preserves the intended task performance for legitimate queries but yields obfuscated, useless gibberish when subjected to adversarial extraction or inversion attacks, breaking the semantic link between the hidden instruction and the observable output.

Key Foundational Work

Teach GPT To Phish (2023) — contributed 2 concept elements to this direction.
AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models (2023) — contributed 1 concept elements to this direction.
Language Model Inversion (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_system_prompt_leakage_afbf2fde,
  title = {system prompt leakage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/system-prompt-leakage/rdr_afbf2fde02554fe797ab8675b30be08f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_system_prompt_leakage_afbf2fde_v1,
  title = {system prompt leakage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/system-prompt-leakage/rdr_afbf2fde02554fe797ab8675b30be08f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake