Bypassing LLM Safety Alignment via Response Prefix Injection

96 concepts62 papers2 proposals

When we train Large Language Models (LLMs) to be safe, we typically teach them a simple behavioral pattern: when a user asks for something harmful, the model should immediately refuse. This refusal usually manifests in the very first tokens of the response—phrases like "I cannot fulfill this request" or "As an AI, I am programmed to be safe."

However, a growing body of research reveals a fundamental flaw in this defense. If an attacker can somehow force the model to begin its response with an affirmative phrase—such as "Sure, here is how to..."—the model's safety guardrails completely collapse. This technique, known as response prefix injection or affirmative prefilling, allows attackers to bypass sophisticated safety alignment with remarkable ease.

To understand why this attack is so devastatingly effective, we have to look under the hood at how autoregressive models process text and why current safety training is often only skin-deep.

The Mechanics of Autoregressive Momentum

At their core, modern language models are autoregressive: they predict the next token based entirely on the sequence of tokens that came before it. This architecture creates a powerful constraint. When a model generates text, it does not just condition its next step on the user's prompt; it conditions heavily on its own emerging response.

As explored by Yiwei Wang et al. in their work on output prefix attacks, LLMs possess an inherent drive for logical coherence. When the beginning of the response buffer is pre-filled with an affirmative phrase, the model treats these tokens as its own prior output. To maintain a consistent narrative and fulfill its primary pretraining objective of sequence completion, the model's internal probability distribution shifts dramatically toward compliance.

This creates what can be thought of as behavioral inertia or statistical momentum. As Jason Vega et al. demonstrate, prepending a partial, compliant response to a harmful query makes it computationally difficult for the system to backtrack. Once the model has committed to a helpful persona at the start of the generation, the statistical gravity of the prefix overrides the safety boundaries established during reinforcement learning. Alexander Wei et al. frame this as a conflict between competing objectives: the model's foundational drive to complete a sequence coherently overpowers its secondary, alignment-enforced drive to refuse.

Standard Safety Path:
[User Prompt: Harmful Request] ---> [Model Internal State: Evaluate Safety] ---> "I cannot fulfill this request."

Prefix Injection Path:
[User Prompt: Harmful Request] + [Injected Prefix: "Sure, here is..."] ---> [Model Internal State: Maintain Coherence] ---> [Harmful Content Completion]

How Prefixes are Injected Across Interfaces

An attacker can force an affirmative prefix into a model's generation buffer through several distinct vectors, depending on how the model is deployed:

  1. API-Level Prefilling: Many commercial APIs allow developers to pass a partial assistant response to guide the model's formatting. As detailed in Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks, this feature can be weaponized to lock the model into a compliant state before it even generates its first token, effectively neutralizing internal refusal triggers.
  2. Chat Template Manipulation: The authors of the ChatBug vulnerability highlight that instruction-tuned models rely on specific control tokens to separate user inputs from assistant responses. By intentionally omitting or malforming these templates, an attacker can "overflow" their prompt into the assistant's response field, tricking the model into treating user-specified text as its own initial output.
  3. Multimodal Priming: The vulnerability extends beyond text. In vision-language and speech models, attackers can optimize continuous inputs (like adversarial image noise or audio signals) to force the model's text decoder into an affirmative state. Research by Raghuveer Peri et al. on audio robustness and Yifan Li et al. on visual vulnerabilities shows that forcing an initial affirmative token via multimodal inputs causes the internal attention mechanisms to prioritize maintaining consistency with that helpful opening over enforcing safety constraints.

The Root Cause: Shallow Alignment

Why does a model fail to self-correct? If a model realizes on token five that it is generating a recipe for something dangerous, why doesn't it simply stop?

The answer lies in how safety alignment is currently implemented. Xiangyu Qi et al. argue that standard safety training is often "shallow," meaning it primarily teaches the model how to behave during the first few tokens of a response. Once an attacker uses a prefix to bypass this initial refusal window, the model frequently reverts to its unaligned base training data to complete the sequence.

Furthermore, Thong Bach et al. identify a structural reason for this superficiality: gradient signal decay during autoregressive training. Because safety objectives are typically trained on full-sentence preferences where the model starts with a clean slate, the model learns to act reactively rather than proactively. It lacks the internal mechanism to evaluate the safety of a prompt independently of the generation trajectory it has already entered. Once the initial tokens are compromised, the model suffers from a form of pseudo-termination or state-locking, as noted by Lijia Lv et al., where it perceives the safety evaluation phase as already complete.

Moving Toward Deep Alignment and Mid-Sequence Recovery

To defend against prefix injection, researchers are shifting away from passive, token-level alignment toward mechanisms that allow models to "catch themselves" mid-sentence.

One promising direction is dual-objective optimization. For instance, Xuandong Zhao proposes training models on augmented trajectories where harmful prefixes are intentionally prepended to safe refusal responses. This explicitly teaches the model a mid-sequence recovery mechanism: even if the generation context begins unsafely, the model learns to pivot back to a refusal.

Similarly, Rongwu Xu et al. introduce the concept of autonomous course-correction using synthetic preference data. By training models to recognize when they have been coerced into an unsafe trajectory, developers can build systems capable of breaking the autoregressive trap—allowing the model to prioritize global safety constraints over local token consistency.

Key Foundational Work

Detecting AI Flaws: Target-Driven Attacks on Internal Faults in Language Models (2024) — contributed 3 concept elements to this direction.
SpeechGuard: Exploring the Adversarial Robustness of Multi-modal Large Language Models (2024) — contributed 3 concept elements to this direction.
EnJa: Ensemble Jailbreak on Large Language Models (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_affirmative_prefix_injection_ff655109,
  title = {affirmative prefix injection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affirmative-prefix-injection/rdr_ff65510941f541fa9068d0ca86aee283},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_affirmative_prefix_injection_ff655109_v1,
  title = {affirmative prefix injection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affirmative-prefix-injection/rdr_ff65510941f541fa9068d0ca86aee283/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake