When we build applications using Large Language Models (LLMs), we typically feed them a mixture of two distinct inputs: developer-defined instructions (such as system prompts or safety guardrails) and untrusted user data (such as a query or a document to summarize). However, to a standard Transformer architecture, these inputs are flattened into a single, undifferentiated sequence of tokens.
As highlighted by Julien Piet et al. in Jatmo: Prompt Injection Defense by Task-Specific Finetuning, this lack of physical or structural separation creates a fundamental vulnerability. Because the model processes instructions and data within the exact same communication channel, malicious user data can easily hijack the model's control flow, causing it to interpret data as high-priority commands. This phenomenon underpins both prompt injection and jailbreaking attacks.
The Illusion of Role Separation
To prevent data from overriding instructions, modern deployment frameworks rely on chat templates that use special tokens (e.g., <|im_start|>system or <|im_start|>user) to establish an instruction hierarchy. The intuition is that the model will learn to respect the authority of the system role over the user role.
However, this boundary is remarkably fragile. In SQL Injection Jailbreak: A Structural Disaster of Large Language Models, J. W. Zhao et al. demonstrate that attackers can simply include these internal structural markers directly within their raw text inputs. By mimicking the model's own control tokens, an attack can effectively "comment out" the original system prompt and force the model into an alternative conversation state.
Furthermore, the way models learn these roles is often superficial. Zihao Wang et al. reveal in The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) that LLMs frequently rely on simple shortcuts—such as the absolute position of a token—rather than a robust semantic understanding of role authority. When these positional expectations are disrupted, the instruction hierarchy collapses. As a result, as shown by Zitong Shi et al. in Don’t Forget the Enjoin: FocalLoRA for Instruction Hierarchical Alignment in Large Language Models, adversarial prompts can easily command the model to ignore its primary directives in favor of malicious user instructions.
Exploitation Across Contexts
This structural vulnerability scales dangerously across different application designs:
- Batch Processing: To reduce inference costs, developers often group multiple independent user queries into a single context window. Murong Yue demonstrates in Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack that this creates a cross-query contamination vector. A single malicious query within the batch can issue global instructions that hijack the model's behavior for all other, completely benign user queries in the same context.
- Multi-Turn Dialogues: Attackers can exploit the model's memory over extended conversations. By actively fabricating dialogue histories to make it appear as though safety warnings have already been bypassed or dismissed, multi-turn jailbreaks can systematically erode a model's caution. This iterative exploitation is detailed by Hua Tang et al. in GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication and by Javad Rafiei Asl et al. in NEXUS: Network Exploration for eXploiting Unsafe Sequences in Multi-Turn LLM Jailbreaks.
- Programmatic Integration: As frameworks increasingly blend natural language prompts with traditional code execution, the attack surface widens. Honghua Dong et al. note in APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts that tight integration between programmatic control flow and untrusted text strings inherently exposes applications to subversion if the underlying prompt boundaries fail.
Mechanistic Defenses
To defend against prompt injections, researchers are moving away from simple keyword filtering and looking directly at the internal mechanics of the Transformer.
Attention and Layer Interventions
When an adversarial prompt successfully hijacks a model, it physically alters how the model allocates attention. Kuo-Han Hung et al. document this in Attention Tracker: Detecting Prompt Injection Attacks in LLMs, identifying a "distraction effect" where specific internal attention heads shift their focus away from the system instructions and toward the malicious user input.
Standard Inference:
[System Prompt] <--- (Attention Heads Focus Here) --- [User Query]
Under Prompt Injection (The Distraction Effect):
[System Prompt] [Malicious Command] <--- (Attention Heads Distracted)
By identifying these highly sensitive heads, Han Jiang et al. developed DeTAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modification, which dynamically re-weights attention during inference to force the model to remain anchored to the core system instructions.
Structural Randomization
Other approaches focus on breaking the predictability of the input layout so that attackers cannot reliably engineer "escape" sequences. Neha Nagaraja et al. introduce this concept in To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt, which dynamically randomizes the delimiters and structural wrapping around user inputs for every single request, isolating untrusted data from the execution context.
Decoding-Time Safeguards
Finally, if an adversarial prompt successfully bypasses input-level defenses, security can still be enforced during token generation. Rather than relying on the model's static weights to refuse a request, frameworks like the one proposed by Jiayou Wang et al. in SecDecoding: Steerable Decoding for Safer LLM Generation use smaller, external guide models to steer the token probabilities in real time. Similarly, Xinyi Zeng et al. demonstrate in ROOT DEFENCE STRATEGIES: ENSURING SAFETY OF LLM AT THE DECODER LEVEL that intervening at the decoding level allows the system to evaluate and correct outputs step-by-step, neutralizing harmful generations even when the prompt itself was compromised.