RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process

Peiran Wang, Xiaogeng Liu, Chaowei Xiao

Abstract

In this study, we introduce RePD, an innovative attack Retrieval-based Prompt Decomposition framework designed to mitigate the risk of jailbreak attacks on large language models (LLMs). Despite rigorous pre-training and fine-tuning focused on ethical alignment, LLMs are still susceptible to jailbreak exploits. RePD operates on a one-shot learning model, wherein it accesses a database of pre-collected jailbreak prompt templates to identify and decompose harmful inquiries embedded within user prompts. This process involves integrating the decomposition of the jailbreak prompt into the user’s original query into a one-shot learning example to effectively teach the LLM to discern and separate malicious components. Consequently, the LLM is equipped to first neutralize any potentially harmful elements before addressing the user’s prompt in a manner that aligns with its ethical guidelines. RePD is versatile and compatible with a variety of open-source LLMs acting as agents. Through comprehensive experimentation with both harmful and benign prompts, we have demonstrated the efficacy of our proposed RePD in enhancing the resilience of LLMs against jailbreak attacks, without compromising their performance in responding to typical user requests.

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains. However, they remain vulnerable to jailbreak attacks, where malicious actors use sophisticated "wrapping" techniques—such as role-playing scenarios or complex encodings—to bypass safety filters and elicit harmful content. Existing defense mechanisms often suffer from high computational costs, excessive refusal of benign prompts (over-defense), or fragility against adaptive attacks.

We propose RePD (Retrieval-based Prompt Decomposition), a novel defense framework that teaches LLMs to "decouple" malicious intent from its surrounding context. By identifying the structural commonalities in jailbreak attempts, RePD enables models to strip away deceptive layers and reject core harmful queries without degrading performance on legitimate requests.

The RePD Framework

The core of RePD is a three-step process designed to unmask hidden attacks using a one-shot learning approach.

The RePD pipeline: from template retrieval to prompt decomposition and final LLM response.
  1. Template Retrieval: Upon receiving a user prompt, RePD performs a similarity search (using a CLIP-based model) against a curated database of known jailbreak templates. This identifies the most likely "wrapper" being used.
  2. One-Shot Learning: Instead of relying on abstract safety instructions, RePD provides the LLM with a concrete example of how a similar template was previously decomposed into its "template" and "core question" components.
  3. Prompt Decomposition: The LLM is tasked with decomposing the current user prompt. By isolating the "core question," the model's internal safety filters are more effectively triggered. If the isolated core is harmful, the model refuses the request.

To counter adaptive attacks, RePD incorporates a Randomization step, where defense instructions are subtly altered with synonyms to create a "moving target." Additionally, a multi-agent variant (RePD-M) can be employed, where one agent specializes in decoupling and another in generating the final response.

Experimental Evaluation

We evaluated RePD across several open-source model families, including Llama-2, Llama-3, Vicuna, and Qwen-2. The evaluation utilized:

  • ToxicChat: A dataset of over 10,000 real-world user interactions for sourcing harmful prompts.
  • Jailbreak Templates: A collection of 74 diverse templates (e.g., DAN, Base64).
  • Benign Questions: 200 safe questions to measure the False Positive Rate (FPR).
  • SALAD Benchmark: A standard for LLM safety evaluation.

Results and Analysis

The experimental results confirm that RePD significantly enhances LLM resilience while maintaining high utility.

ASR and FPR metrics across different model families and sizes, comparing RePD and RePD-M.

Key findings include:

  • Significant Attack Mitigation: RePD reduced the Attack Success Rate (ASR) by an average of 87.2% across various models.
  • Low Over-defense: The framework maintained a low False Positive Rate (FPR) of approximately 8.2%, ensuring that legitimate user queries are rarely blocked.
  • Robustness: Even against gradient-based adaptive attacks like GCG, RePD kept the ASR below 10%.
  • Scalability: The "decoupling" strategy proved effective even for smaller models that typically struggle with zero-shot safety prompts, as the one-shot retrieval provides a clear "cheat sheet" for identifying malicious intent.

Conclusion

RePD offers a robust, model-agnostic defense against jailbreak attacks by forcing LLMs to explicitly identify the core intent of a prompt before responding. While the retrieval process and one-shot examples introduce some latency, the significant gains in safety and the low impact on utility make it a promising approach for securing real-world LLM deployments. Future work will focus on optimizing the retrieval process and extending the framework to defend against non-template-based attack vectors.

Summary updated

Abstract

In this study, we introduce RePD, an innovative attack Retrieval-based Prompt Decomposition framework designed to mitigate the risk of jailbreak attacks on large language models (LLMs). Despite rigorous pre-training and fine-tuning focused on ethical alignment, LLMs are still susceptible to jailbreak exploits. RePD operates on a one-shot learning model, wherein it accesses a database of pre-collected jailbreak prompt templates to identify and decompose harmful inquiries embedded within user prompts. This process involves integrating the decomposition of the jailbreak prompt into the user’s original query into a one-shot learning example to effectively teach the LLM to discern and separate malicious components. Consequently, the LLM is equipped to first neutralize any potentially harmful elements before addressing the user’s prompt in a manner that aligns with its ethical guidelines. RePD is versatile and compatible with a variety of open-source LLMs acting as agents. Through comprehensive experimentation with both harmful and benign prompts, we have demonstrated the efficacy of our proposed RePD in enhancing the resilience of LLMs against jailbreak attacks, without compromising their performance in responding to typical user requests.
BibTeX
@inproceedings{wang2025repd,
  author = {Peiran Wang and Xiaogeng Liu and Chaowei Xiao},
  title = {{RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=S9piGJ1RB8},
}
Figure