TL;DR
This paper introduces CEIPA, a novel framework for systematically analyzing and exploiting vulnerabilities in Large Language Models (LLMs) through incremental prompt attacks. By gradually modifying prompts across word, sentence, character, and combined levels, CEIPA provides counterfactual explanations for LLM failures, highlighting how subtle changes can bypass defenses and lead to harmful outputs. This research is crucial for understanding LLM security and developing more robust defenses.
What Problem Does This Paper Address?
The rapid integration of powerful LLMs like GPT-4 and LLaMA-2 into various applications has introduced significant security challenges. These models are susceptible to various attacks, including prompt injection, which can lead to the generation of harmful, unethical, or privacy-breaching content. Existing research has explored some attack vectors, but a systematic and explainable method to understand why these attacks succeed and how LLMs' defenses can be incrementally bypassed has been lacking. This paper aims to fill that gap by providing a structured approach to probe LLM vulnerabilities and offer insights into their defense mechanisms.
Methods
The core of this research is the Counterfactual Explainable Incremental Prompt Attack (CEIPA) framework. CEIPA operates by starting with a weak attack prompt that initially fails to compromise the LLM. This prompt is then incrementally modified across four distinct levels:
- Word Level: Focuses on replacing or modifying individual words. This involves identifying "replaceable" words (excluding proper nouns, pronouns, prepositions, etc.) and using techniques like synonym substitution, often leveraging GPT-4 for synonym generation. The impact of these word changes is measured using BERT-based similarity scores.
- Sentence Level: Involves summarizing sub-sentences of the prompt and then generating similar sentences to replace them. This process uses a Summarize-and-Rewrite (S&R) technique and relies on LLM judgments and BLEU scores to ensure the mutated sentence maintains semantic similarity while potentially altering the attack's effectiveness.
- Character Level: Introduces subtle spelling errors or character substitutions within important words. Tactics include inserting spaces, deleting characters, swapping characters, or substituting visually similar characters (e.g., 'o' with 'O', 'l' with '1'). These mutations aim to create "unknowns" that test the LLM's robustness to minor input perturbations.
- Word/Char Level: Combines word-level strategies (like synonym substitution) with character-level modifications.
The framework systematically applies these mutations in sequence, observing the LLM's response at each step. The goal is to identify the "transition points" where a failed attack becomes successful, providing counterfactual explanations for the LLM's behavior.
Datasets
The study utilizes several datasets for its experiments:
- Jailbreak Chat dataset: 70 unique attacking prompts were selected from this dataset, combined with 100 distinct attacking questions, resulting in 7,000 unique attack prompts for the jailbreak task.
- Tensor Trust datasets: Two specific datasets from Tensor Trust were used for the system prompt extraction and hijacking tasks, including both system prompt and attacking prompt datasets.
The paper does not introduce new datasets but rather leverages existing ones for evaluating its CEIPA framework.
Key Insights and Ideas
- Incremental Mutation is Key: The study demonstrates that gradually increasing the complexity of prompt modifications is highly effective in bypassing LLM defenses. Small, incremental changes are often more successful than large, abrupt ones.
- Sentence and Word Levels are Potent: The sentence and word-level mutations proved to be the most effective in increasing attack success rates, suggesting that LLMs are particularly sensitive to changes at these granularities.
- Char/Word Mutations Can Be Defensive: Interestingly, the char/word level mutations, while sometimes effective, also showed potential to act as a defensive mechanism. In some cases, these mutations could dilute the toxicity of attack prompts, suggesting a novel defense strategy.
- LLMs Prioritize Context: The research suggests that LLMs might rely more heavily on sentence-level context than individual words for decision-making. This implies that subtle changes to sentence structure or meaning can have a more significant impact than altering single words.
- Verbs and Adjectives are Pivotal: Analysis of word-level attacks revealed that verbs and adjectives are frequently selected as the most influential words for successful prompt transitions, indicating their critical role in shaping LLM responses.
Key Results
The CEIPA framework significantly enhanced attack success rates across various LLMs (GPT-3.5, LLaMA2-13B, Gemma, Guanaco) and tasks (Jailbreak, System Prompt Extraction, Hijacking).
- Jailbreak Task: Mutations at the word and sentence levels elevated the Attack Success Rate (ASR) from a baseline CLEAN ASR of 0.29% to approximately 0.50% on GPT-3.5. The sentence level showed a superior success rate, peaking around the 12th round.
- System Prompt Extraction: Similar trends were observed, with sentence and word levels demonstrating the highest efficacy. A concave pattern was noted at the word level, peaking around the 22nd round before declining, attributed to shorter prompts losing effectiveness as synonyms were exhausted.
- Prompt Hijacking: The improvement in attack success was more modest due to the demanding success criteria (requiring an explicit "Access Granted" response). CEIPA enhanced CLEAN ASR by a maximum of 6% for GPT-3.5 and 4% for LLaMA2-13B.
- Transfer Success: Attack prompts effective against GPT-3.5 or Guanaco sometimes struggled against LLaMA2-13B and Gemma, indicating these models possess stronger security against certain attack vectors.
Limitations and Future Work
- Limited LLM Scope: The current study primarily focuses on a selected group of LLMs. Future work aims to broaden this scope to include more diverse models and tasks, such as acquiring private information.
- Optimization of Techniques: The authors plan to explore more effective methods to optimize their attack processes and potentially surpass the capabilities of existing tools like BERT-Attack.
- Refinement of Defense Strategies: The insight that char/word mutations can act defensively suggests further research into developing specific mutation-based defense mechanisms.
- Deeper Linguistic Analysis: The identification of verbs and adjectives as pivotal words warrants further investigation into the linguistic and contextual aspects that drive successful attacks.
- Comprehensive Testing: The researchers intend to conduct comprehensive testing across a wide range of tasks and models to systematically document vulnerabilities and provide feedback to LLM developers.
Summary updated