Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models

Xiao Liu, Liangzhi Li, Tong Xiang, Fuying Ye, Lu Wei, Wangyue Li, Noa GarciaarXiv 2024 · DOI · arXiv

Introduction

As Large Language Models (LLMs) become increasingly integrated into daily applications, developers have implemented rigorous safety mechanisms, such as Reinforcement Learning from Human Feedback (RLHF), to prevent the generation of harmful content. However, most existing adversarial attacks rely on explicit "jailbreak" prompts or universal patterns that are easily identified and mitigated. This paper introduces Imposter.AI, a novel framework that exploits a critical vulnerability: hidden intentions. By decomposing a malicious goal into a series of seemingly benign sub-questions, Imposter.AI mimics human conversational strategies to bypass safety alignments in state-of-the-art models like GPT-4.

Methodology

The Imposter.AI pipeline is an automated, multi-stage process designed to obfuscate malicious intent through multi-turn dialogue. The process consists of five key stages:

  1. Knowledge Extraction: An uncensored LLM is used to generate a high-level roadmap (keypoints) for answering a harmful question.
  2. Harmful Question Decomposition: An "Agent LLM" (e.g., GPT-4) transforms these keypoints into a sequence of sub-questions, sorted from most benign to most sensitive to "prime" the target model.
  3. Question Toxicity Reduction: Sub-questions are rephrased using techniques such as Perspective Change (e.g., asking about historical criminal methods) and Intent Reversion (e.g., asking how to prevent a harm to indirectly learn how to cause it).
  4. Response Harmfulness Enhancement: The attack prompts for "Fictional Scenarios" or "Historical Examples" to ensure the target LLM provides detailed, actionable information.
  5. Summarization: Finally, the target LLM is asked to summarize the entire conversation into a step-by-step manual, consolidating fragmented data into a single harmful response.
The Imposter.AI pipeline: from harmful question decomposition to response summarization.

Experimental Setup

The researchers evaluated Imposter.AI using the HarmfulQ dataset, which contains 200 explicit harmful questions across 16 categories, including terrorism and weapons. The framework was tested against GPT-3.5-turbo, GPT-4, and Llama2-13b, and compared to established baselines such as AIM (a manual jailbreak) and GCG (an optimization-based attack).

Results and Analysis

Imposter.AI demonstrated significant effectiveness across multiple models, particularly outperforming baselines on GPT-4.

  • Superior Performance: On GPT-4, Imposter.AI achieved a Harmfulness score of 4.38/5 and an Executability score of 3.14/5, significantly higher than GCG and AIM.
  • Persistence: Unlike manual jailbreaks like AIM, which saw a sharp decline in effectiveness as OpenAI patched its filters, Imposter.AI's success rate remained stable, highlighting the difficulty of filtering natural conversational strategies.
  • The Usability-Security Trade-off: While Llama2-13b showed high resistance to the attack, it often did so through "over-refusal," rejecting even benignly rephrased versions of questions. This suggests that extreme safety alignments can significantly degrade a model's utility for legitimate users.
Comparative performance of Imposter.AI against baselines (AIM, GCG) across GPT-4, GPT-3.5-turbo, and Llama2-13b.

Conclusion

The success of Imposter.AI reveals that current LLM safety filters suffer from "contextual blindness," evaluating safety on a per-turn basis rather than analyzing the global intent of a conversation. To counter such multi-turn adversarial strategies, future defense mechanisms must evolve to understand the overarching goals of a dialogue history. The findings also underscore the need for a better balance between model safety and helpfulness.

Summary updated

Abstract

With the development of large language models (LLMs) like ChatGPT, both their vast applications and potential vulnerabilities have come to the forefront. While developers have integrated multiple safety mechanisms to mitigate their misuse, a risk remains, particularly when models encounter adversarial inputs. This study unveils an attack mechanism that capitalizes on human conversation strategies to extract harmful information from LLMs. We delineate three pivotal strategies: (i) decomposing malicious questions into seemingly innocent sub-questions; (ii) rewriting overtly malicious questions into more covert, benign-sounding ones; (iii) enhancing the harmfulness of responses by prompting models for illustrative examples. Unlike conventional methods that target explicit malicious responses, our approach delves deeper into the nature of the information provided in responses. Through our experiments conducted on GPT-3.5-turbo, GPT-4, and Llama2, our method has demonstrated a marked efficacy compared to conventional attack methods. In summary, this work introduces a novel attack method that outperforms previous approaches, raising an important question: How to discern whether the ultimate intent in a dialogue is malicious?
BibTeX
@misc{liu2024imposter,
  author = {Xiao Liu and Liangzhi Li and Tong Xiang and Fuying Ye and Lu Wei and Wangyue Li and Noa Garcia},
  title = {{Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models}},
  year = {2024},
  eprint = {2407.15399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.15399},
}
Figure