Proxy Barrier: A Hidden Repeater Layer Defense Against System Prompt Leakage and Jailbreaking

Pedro Schindler Freire Brasil Ribeiro, Iago Alves Brito, Rafael Teixeira Sousa, Fernanda Bufon Färber, Julia Soares Dollis, Arlindo Rodrigues Galvão FilhoEMNLP 2025

TL;DR

This paper introduces Proxy Barrier (ProB), a novel defense mechanism for Large Language Models (LLMs) that acts as a "honeypot" by interposing a simple "repeater" LLM. ProB effectively blocks prompt injection and jailbreaking attacks by ensuring user inputs are faithfully echoed by the proxy before reaching the main LLM, achieving over 98% defense effectiveness in some configurations.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful but vulnerable to "prompt injection" and "jailbreaking" attacks. These attacks trick LLMs into bypassing safety protocols, revealing sensitive information (like system prompts), or performing restricted/unsafe actions. Existing defenses often require privileged access to the LLM's internals, struggle with new attack types, or degrade performance. This paper tackles the need for a simple, API-level defense that is effective against a wide range of attacks, works in black-box settings, and maintains response quality.

Methods

The core innovation is the Proxy Barrier (ProB). It works by inserting a proxy LLM between the user and the target LLM. This proxy LLM is specifically instructed to faithfully repeat the user's input verbatim.

Here's the workflow:

  1. User Input: A user submits a query.
  2. Proxy Processing: The query goes to the proxy LLM, whose sole job is to echo it exactly.
  3. Comparison: The output of the proxy LLM is compared to the original user input.
  4. Decision:
    • If the proxy output faithfully matches the input, the query is deemed benign and forwarded to the main (safeguarded) LLM.
  5. If there's any discrepancy (indicating the proxy was manipulated to deviate from a simple echo), it's flagged as a potential attack, and the query is blocked.

Crucially, ProB doesn't need access to the target LLM's weights or system prompts. It can be deployed at the API level. The paper also explores using smaller, more vulnerable models as proxies, which are more likely to "fail" (deviate from echoing) on adversarial inputs, thus acting as a more effective honeypot.

Datasets

  • Unnatural Instructions dataset (Honovich et al., 2022): Used for generating benign queries and for fine-tuning the proxy model.
  • WildChat dataset (Zhao et al., 2024): Used for fine-tuning and testing the stability of few-shot examples.
  • Jailbreakv28k benchmark (Luo et al., 2024): A curated dataset of 280 diverse jailbreak prompts used for evaluating defense against jailbreaking.
  • Custom attack datasets: The authors created datasets for prompt leakage, including variations of existing prompts and multi-turn scenarios.

Key Insights and Ideas

  • The "Honeypot" Principle: A simple, vulnerable proxy LLM is more effective at detecting attacks than a highly capable one. By making the proxy easier to break, it becomes a better early warning system.
  • Whitelist vs. Blacklist: ProB operates on a "whitelist" principle – only inputs that can be faithfully repeated are allowed. This is more robust than blacklisting specific attack patterns, which can be easily circumvented.
  • API-Level Defense: The ability to deploy ProB solely at the API level is a significant advantage for practical adoption, especially for third-party LLM services.
  • Counter-intuitive Fine-tuning: Fine-tuning the proxy LLM to be more susceptible to attacks (rather than more robust) paradoxically enhances the overall defense by making it more likely to trigger the blocking mechanism.
  • Model Scaling Matters: For ProB to be effective, the proxy LLM should ideally be of comparable or lesser capability than the safeguarded LLM. High-capacity models often fail to act as effective proxies because they faithfully repeat everything.

Key Results

  • High Defense Effectiveness: ProB achieved up to 98.8% defense effectiveness against prompt leakage attacks.
  • Jailbreaking Reduction: ProB significantly reduced successful jailbreaks, with some models achieving perfect benchmark scores (0% attack success rate).
  • Minimal Performance Overhead: When using a smaller proxy model, ProB offers a cost-efficient solution while maintaining or increasing security.
  • Robustness: ProB demonstrated robustness across various LLM families (Llama, Gemini, GPT-4) and attack types, including multi-turn scenarios.
  • False Positive Rate: While some configurations showed high false positive rates (blocking benign queries), the paper demonstrates strategies (like using Llama-3-405B as a proxy) to achieve a good balance between security and usability, with rates as low as 19% for some models.

Limitations and Future Work

  • Scope of Attacks: ProB is primarily designed to block direct prompt leakage and injection attacks that aim to extract system prompts via output. It does not address more advanced techniques like LLM inversion or distribution tracing, which require white-box access.
  • False Positives: While improved, false positives (blocking benign queries) can still degrade user experience. The paper notes that multi-turn conversations can help ground the proxy and reduce this.
  • Prompt Sensitivity: The few-shot prompt for the proxy LLM can be sensitive to minor changes in formatting or wording, requiring careful prompt engineering.
  • Future Work: The authors suggest exploring more sophisticated comparison metrics for alternative reversible transformations (like translation or word shuffling) and further investigating fine-tuning strategies to enhance both security and utility. They also highlight the potential for ProB to be generalized to other LLM-based security challenges.

Summary updated

BibTeX
@inproceedings{ribeiro2025proxy,
  author = {Pedro Schindler Freire Brasil Ribeiro and Iago Alves Brito and Rafael Teixeira Sousa and Fernanda Bufon Färber and Julia Soares Dollis and Arlindo Rodrigues Galvão Filho},
  title = {{Proxy Barrier: A Hidden Repeater Layer Defense Against System Prompt Leakage and Jailbreaking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure