Abstract
Prompt injection attacks, where an attacker injects a prompt into the original one to hijack a Large Language Model's (LLM) behavior, represent a critical security threat. Existing attacks primarily focus on inference-time techniques, treating the LLM as a static target. In this work, we introduce a more foundational attack vector: poisoning the LLM's alignment process to amplify the success of future prompt injection attacks. Specifically, we propose PoisonedAlign, a method that strategically creates poisoned alignment samples to corrupt an LLM's alignment dataset. Our experiments across five LLMs and two alignment datasets show that when even a small fraction of the alignment data is poisoned, the resulting model becomes substantially more vulnerable to a wide range of prompt injection attacks. Crucially, this vulnerability is instilled while the LLM's performance on standard capability benchmarks remains largely unchanged, making the manipulation difficult to detect through automated, general-purpose performance evaluations. The code for implementing the attack is available at https://github.com/Sadcardation/PoisonedAlign.
Introduction
Large Language Models (LLMs) are increasingly integrated into complex workflows where they process untrusted data, such as user reviews or web search results. This integration has exposed a significant vulnerability known as prompt injection, where malicious instructions embedded in data trick the model into ignoring its original task. While most research has focused on finding the perfect "separator" string at inference time to bypass defenses, these heuristic methods are hitting a ceiling as models become more robust.
This paper shifts the focus to the alignment stage (Supervised Fine-Tuning and Preference Alignment). We ask: Can we corrupt the model's training so that it becomes inherently "eager" to follow injected instructions? By introducing PoisonedAlign, we demonstrate that the alignment process itself can be turned into a weapon to weaken the model's defenses systematically.
Methodology: The PoisonedAlign Framework
PoisonedAlign is a systematic approach to data poisoning during the alignment phase. Unlike traditional backdoor attacks that require a specific, static "trigger" word, PoisonedAlign instills a systemic bias by exploiting the general structure of prompt injection.
The method creates poisoned samples for two common alignment techniques:
- Supervised Fine-Tuning (SFT): The attacker creates samples where a target prompt is merged with an injected prompt using a separator. The model is trained to output the response corresponding to the injected task.
- Preference Alignment (e.g., DPO): The attacker constructs triples where the model is taught to prefer the response to the injected prompt over the response to the original target prompt.
Essentially, the model is "aligned" to believe that when it sees a conflict between an initial instruction and a later one following a separator, the later one is the correct one to follow.
Experimental Setup
We utilized two widely used alignment datasets: HH-RLHF (Helpful and Harmless) and ORCA-DPO. For evaluation, we employed a 7x7 task matrix involving seven natural language tasks, including sentiment analysis, text summarization, and hate detection. This allowed us to test how poisoning on one set of tasks generalizes to others across five different LLMs: Llama-2, Llama-3, Gemma, Falcon, and GPT-4o mini (via its fine-tuning API).
Key Results and Insights
Our experiments demonstrate that PoisonedAlign significantly increases model vulnerability while remaining remarkably stealthy.
Significant Attack Boost
With only a 10% poisoning rate in the ORCA-DPO dataset, the Attack Success Value (ASV) of a Combined Attack against Llama-3-8b-Instruct increased by an average of 0.33. As shown in the figure below, the poisoned model becomes significantly more susceptible to various injection strategies—including naive attacks and escape characters—compared to models that underwent clean alignment.
Stealth and Generalization
- Stealthiness: A poisoned model's scores on standard benchmarks like MMLU or GSM8K remain virtually unchanged (usually within a 2% margin), meaning a developer might deploy a compromised model without realizing its security has been gutted.
- The "Helpfulness" Paradox: Interestingly, stronger models with better instruction-following capabilities (like Llama-3) were found to be more vulnerable to this poisoning. Their inherent drive to be "helpful" makes them more susceptible to the corrupted alignment.
- Defense Failure: We tested state-of-the-art backdoor detection methods like BEAT and BAIT. These largely failed to distinguish between "cleanly" aligned models and those poisoned by PoisonedAlign because the latter does not rely on a fixed, detectable trigger.
Conclusion
PoisonedAlign reveals a foundational vulnerability in the LLM lifecycle. By poisoning a small fraction of the alignment data, attackers can instill a systemic bias that makes models inherently prone to prompt injection. This attack is particularly dangerous because it bypasses standard quality control and generalizes across different attack styles. Future research must focus on developing structural defenses that can identify these systemic biases during the alignment process.
Summary updated