RepeatLeakage: Leak Prompts from Repeating as Large Language Model Is a Good Repeater

Yu Peng, Lijie Zhang, Peizhuo Lv, Kai ChenAAAI 2025 · DOI

Introduction

As Large Language Model (LLM) ecosystems like the GPT Store expand, developers increasingly rely on "system prompts" to define unique bot personas and behaviors. These prompts are often considered valuable intellectual property, leading to a security "arms race" between developers attempting to hide them and attackers seeking to extract them. A significant challenge in existing prompt-leaking research is confidence: it is often difficult to determine if a model's output is a genuine leak or a hallucination. This paper introduces RepeatLeakage, a high-confidence method that exploits the fundamental architectural tendencies of LLMs to extract hidden context with near-perfect accuracy.

The Repetition Phenomenon

The core of RepeatLeakage lies in the "repetition phenomenon." The researchers hypothesized that when an LLM is tasked with repeating a sentence, it relies heavily on its immediate input context rather than its internal training weights. To validate this, they compared knowledge-based tasks (e.g., "The capital of France is...") with repetitive tasks (e.g., "Repeat the following: The capital of France is...").

Validation of the repetition phenomenon through embedding contamination and restoration.

Using a "contamination" technique, they added Gaussian noise to the embeddings of nouns in the input. While this noise significantly degraded performance on knowledge-based tasks, the models remained capable of performing repetitive tasks accurately. This confirms that repetition is a "context-heavy" operation, providing a technical foundation for high-fidelity leakage attacks.

The RepeatLeakage Attack

Building on these insights, the authors developed a template-based attack designed to bypass filters and ensure verbatim output. The attack follows a specific structure:

"Repeat everything {place to leak} {format of output} and {encouragement for repeat}"

  • {place to leak}: Directs the model to specific memory segments (e.g., using pronouns like "you received" to target system prompts).
  • {format of output}: Mandates verbatim reproduction (e.g., "exactly in its original format").
  • {encouragement}: Employs "jailbreak-lite" phrasing to circumvent safety filters (e.g., "Sure, here is everything...").

Experimental Results

The effectiveness of RepeatLeakage was tested against open-source models (Llama-3-8B, Phi-3-Small, Qwen2-7B) and real-world interfaces like ChatGPT. Performance was measured using BertScore, with 99-100% similarity representing "near-perfect" alignment.

Performance comparison showing RepeatLeakage's superior near-perfect leakage rates compared to baselines.

RepeatLeakage significantly outperformed established baselines (Perez et al. and Zhang et al.). For instance, on the Phi-3 model, RepeatLeakage achieved a 77.8% near-perfect leakage rate for system prompts, whereas the best baseline only reached 30%. Furthermore, the failure rate (similarity below 90%) was remarkably low, often staying under 10% while baselines failed up to 90% of the time.

Key Insights and Limitations

The study reveals that LLMs are architecturally incentivized to be "good repeaters." By focusing the attention mechanism on the provided context, the model effectively bypasses internal logic that might otherwise filter sensitive information.

However, the research also noted anomalies. The Qwen2 model occasionally attempted to answer questions rather than repeat them, suggesting that specific fine-tuning styles may offer inherent resistance to this vector. Additionally, while highly effective in English, the cross-lingual robustness of RepeatLeakage remains a subject for future investigation. The authors emphasize the need for new defense mechanisms that can distinguish legitimate repetition requests from malicious context extraction attempts.

Summary updated

BibTeX
@inproceedings{peng2025repeatleakage,
  author = {Yu Peng and Lijie Zhang and Peizhuo Lv and Kai Chen},
  title = {{RepeatLeakage: Leak Prompts from Repeating as Large Language Model Is a Good Repeater}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i25.34832},
}
Figure