Language models can learn a new task from a few worked examples placed in the prompt. This is called in-context learning (ICL), and it involves no weight updates. It is also fragile. Anwar et al. showed that changing a single token in one demonstration can force a transformer's output to any target the attacker chooses. This matters most when demonstrations come from untrusted sources, such as retrieved documents or user-supplied examples. Adversarial training is the standard fix: train the model on worst-case corrupted prompts. It has two problems here. Its inner attack loop runs a forward and backward pass through the whole network many times, so it is expensive. It also tends to lower accuracy on clean inputs.
Interpretability research offers a way around both problems. Sia et al. found that each model has a fairly narrow band of middle layers where it "recognizes" the task. After that band, the query no longer needs to look at the demonstrations at all; cutting that attention costs no accuracy and saves 45–50% of inference compute (Where does In-context Learning Happen in Large Language Models?). We propose to enforce that cut. Once the query is blinded to the demonstrations above the recognition layer, any attack on the demonstrations can only reach the output by moving the query's internal state at that one layer. The attack surface is therefore a small, fixed set of vectors at a known depth. Adversarial training then only needs to stop demonstration perturbations from moving those vectors. Its inner attack loop backpropagates through the lower part of the network only and never through the output layers, and the weight update can leave the upper layers, where the task is executed, untouched.
Core Claim
Fix a recognition layer $L^*$, found with Sia's context-masking diagnostic, and mask attention from query and generated positions to demonstration tokens above $L^*$. The claim has three parts.
First, under this masking, whether a demonstration hijack succeeds is fully mediated by how far it moves the query-position hidden states at $L^*$. Patching the clean states back in should undo the attack.
Second, adversarial training that only penalizes displacement at $L^*$ ("recognition-stage adversarial training") should reach hijack robustness comparable to adversarial training that perturbs demonstrations and backpropagates the full output loss. It should do so at roughly $L^*/L$ of the inner-loop cost, where $L$ is the number of layers. For Llama-3.1-8B, whose plateau sits near layer 14 of 32, that is about 44%.
Third, because execution layers are frozen, the method should pay a smaller clean-accuracy tax on held-out ICL tasks and on general benchmarks.
The training objective is
$$\min_{\theta_{\le L^*}} \; \mathbb{E}\Big[\mathcal{L}_{\text{task}}(f_\theta(p)) + \lambda \max_{\delta \in \Delta_k} \big\| h_{L^*}(p+\delta) - \mathrm{sg}\big(h_{L^*}(p)\big)\big\|^2\Big].$$
The symbols are as follows. $p$ is a clean few-shot prompt. $\delta$ is a perturbation to the embeddings of at most $k$ demonstration tokens, drawn from the allowed set $\Delta_k$. $h_{L^*}$ stacks the query-position hidden states at layer $L^*$. $\mathrm{sg}$ is stop-gradient. $\theta_{\le L^*}$ are the parameters at or below $L^*$. $\lambda$ weights the robustness term.
The inner maximization never runs layers above $L^*$, and it involves no decoding. The alignment form of the loss follows the clean/adversarial feature alignment that Robustness Tokens used on frozen vision transformers.
One point explains why the defense has to sit below $L^*$ and not above it. In-Context Learning Creates Task Vectors found that a patched task vector overrides contradicting text. A hijack that moves the query state into, say, an "antonym" region therefore looks identical at $L^*$ to a genuine antonym prompt. Upper layers cannot be trained to resist it without destroying ICL itself. The only workable place to defend is the recognition stage, which must be made insensitive to small corruptions of individual demonstrations.
Relation to Prior Work
Several papers motivate the proposal by showing the gap. Anwar et al. (Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens) demonstrate the one-token hijack and show that adversarial training with $k_{train}=1,3$ generalizes to $k_{test}=7$ with little clean-error cost. Their study is behavioral, on synthetic linear regression, and uses full-depth training. We move the threat model to real language tasks and locate where the defense must act. Kumano et al. (Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners) prove that adversarial pretraining makes one-layer linear attention amplify robust features. They also report that the resulting learners need more demonstrations to reach clean accuracy ("sample-hungry"). They say nothing about multi-layer circuits. The two papers disagree on whether ICL robustness costs clean accuracy, and our shot-count curves are designed to test that question in real models.
Two papers supply methods we reuse. Sia et al. provide the masking diagnostic and the redundancy result that makes the mediation argument possible. Kharlapenko et al. (Scaling Sparse Feature Circuits For Studying In-Context Learning) find task-detection features feeding task-execution features in Gemma 1 2B. Sun et al. (Interpret and Improve In-Context Learning via the Lens of Input-Label Mappings) locate input-label mapping in about 5% of heads and show that tuning only those heads leaves MMLU nearly unchanged. Together these let us check which features an attack actually moves, and they provide a narrower outer update to try.
The closest competitors are latent adversarial training methods. Defending Against Unforeseen Failure Modes with Latent Adversarial Training perturbs hidden layers chosen heuristically. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs uses four evenly spaced layers and reports about 700x less compute than R2D2. Neither method studies ICL. Both backpropagate an output loss through every layer above the perturbation point. Neither masks the attention paths that would let corrupted demonstration tokens bypass the perturbed layer.
ER-CAT (Understanding and Improving Continuous LLM Adversarial Training via In-context Learning Theory) uses ICL theory only to regularize input-embedding singular values for jailbreak defense. It argues that vulnerability lives at the embedding level. The latent adversarial training papers argue it lives deeper. Our layer ablation tests where demonstration-hijack vulnerability actually sits.
Research Program
The work has four stages.
The first stage measures where attacks act. For Llama-3.1-8B, Mistral-7B, and Gemma 1 2B, we locate $L^*$ per task family with layer-from context masking. We then attack the demonstrations, with and without masking. The attacks are continuous embedding perturbations, GCG-style discrete substitutions restricted to demonstration tokens, and single-label flips. We record each attack's displacement at $L^*$ and patch the clean states back in. On Gemma we also check whether the attack moves detection features, execution features, or both.
The second stage trains the defense. We apply recognition-stage adversarial training with LoRA, restricted to layers at or below $L^*$. A second variant updates only the Sun et al. minority heads.
The third stage measures compute. We count forward and backward passes and GPU-hours, following the accounting used by Targeted latent adversarial training.
The fourth stage tests adaptive attacks. These are attacks that know the masking and the training. They include attacks that try to delay recognition past $L^*$, which under masking should turn a hijack into a measurable drop in accuracy rather than a controlled output.
Baselines
The naive reference is the undefended model with standard prompts. It sets the hijack success rate and the clean accuracy-versus-shots curve. A second reference is context masking alone with no training, which isolates whether the architectural cut changes vulnerability by itself.
The strongest existing competitor is Targeted latent adversarial training. It is the method most likely to win, because it already beats token-space training by orders of magnitude in compute while keeping utility: the paper reports an MMLU drop "typically less than 1%" and "approximately 700x less compute" than R2D2 on Llama-2-7B. We will retrain it on exactly our ICL training prompts. We will use its four evenly spaced layers, give it the same LoRA rank, data, and number of optimizer steps, and match its hyperparameter search budget to ours.
Full-depth demonstration-space adversarial training is the direct analogue of Anwar et al. It perturbs demonstration embeddings and uses the output loss, and it is the robustness ceiling our cheaper method must approach. ER-CAT with the same data represents embedding-level defenses.
Lacuna has no reported result for demonstration-hijack success rates on real LLMs for any of these methods. The baselines therefore must be measured, not quoted.
Beating the competitors means two things. Our method should have hijack success at or below Targeted latent adversarial training and full-depth training at $k=1,3,7$. It should also use fewer inner-loop passes, and show no larger clean-accuracy drop on held-out tasks or MMLU.
The key ablations hold everything else fixed and change one ingredient. The first applies the same objective at $L^*$ without masking, which tests whether masking is what makes the bottleneck complete. The second applies it at a mismatched layer, $L^* \pm 6$. The third replaces the alignment loss with a full-depth output loss, which isolates the savings from truncating the inner loop. The fourth updates all layers instead of only those at or below $L^*$.
Why not just…?
Why not just sweep the latent adversarial training layer and take the best one? This is the most serious alternative. A sweep might land on $L^*$ anyway. The sweep costs one full training run per candidate layer, while the masking diagnostic costs only inference. The more important difference is that latent adversarial training without masking leaves attention paths above the perturbed layer through which corrupted demonstrations can still act. The masking-on/masking-off ablation at an identical layer settles this. If the swept-layer baseline, run without masking, matches our robustness and compute, then mechanistic localization adds only a faster way to choose the layer, and we will report it that way.
Why not just ensemble over subsets of demonstrations at test time? Majority voting over leave-some-out prompts needs no training and could defeat one-token hijacks. It is a real threat at $k=1$. It multiplies inference cost by the number of subsets, and it breaks once corruptions are spread across demonstrations so that most subsets contain one. We include it as a cost-matched baseline at $k=1,3,7$ and against attacks that spread corruptions. Our expectation, which the experiment can refute, is that it degrades faster than trained recognition-stage robustness as $k$ grows.
Why not just filter suspicious inputs or warn the model in the system prompt? Baseline Defenses for Adversarial Attacks Against Aligned Language Models shows that perplexity filters and retokenization reject or garble legitimate unusual queries. Continuous perturbations and natural-looking label flips do not raise perplexity at all. The task-vector conflict results suggest that a written instruction cannot override a corrupted internal task representation. We will include a perplexity filter and an instruction-prefixed prompt as baselines and report their clean false-rejection rate.
Why not just run adversarial LoRA on all layers? Sun et al. report a 21% MMLU drop under full fine-tuning, versus -0.03% for the targeted heads. That points to collateral damage from broad updates, though their setting was not adversarial. The all-layer outer-update ablation measures the clean-accuracy cost directly.
Proposed Experiments
The evaluation tasks are the function-vector token tasks from the Kharlapenko et al. suite; SST-2, RTE, QQP, and ETHOS few-shot classification; and FLORES 5-shot translation. Synthetic 20-dimensional regression on GPT-2-style models serves as a sanity check against Anwar et al. Training and test tasks are disjoint.
The metrics are hijack success rate at $k\in\{1,3,7\}$, clean ICL accuracy as a function of the number of shots (to test sample hunger), MMLU and AlpacaEval length-controlled win rate for general utility, and inner-loop forward and backward passes and wall-clock time. As reference points, Sun et al. report "-0.03%" MMLU for head-restricted tuning, and Geva-style task vectors recover 80–90% of multi-shot accuracy.
There are three decisive checks.
The first is mediation. Under masking, patching clean $L^*$ states into attacked runs should restore the clean output in nearly all cases, and $L^*$ displacement should predict hijack success. If either fails, the core premise is wrong.
The second is a cost-matched comparison of robustness against clean accuracy. Plot hijack success against clean held-out accuracy at equal pass budgets for every method. The proposal succeeds only if recognition-stage training reaches the frontier of Targeted latent adversarial training or improves on it.
The third is sample hunger. If the clean-accuracy gap to the undefended model shrinks with more shots, as Kumano et al. would predict, the tax is real but bounded. If it does not appear, that supports Anwar et al.'s minimal-tradeoff finding at scale. Either outcome resolves the disagreement between them for multi-layer models.
References
- Where does In-context Learning Happen in Large Language Models?
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- In-Context Learning Creates Task Vectors
- Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens
- Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners
- Scaling Sparse Feature Circuits For Studying In-Context Learning
- Interpret and Improve In-Context Learning via the Lens of Input-Label Mappings
- Defending Against Unforeseen Failure Modes with Latent Adversarial Training
- Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
- Understanding and Improving Continuous LLM Adversarial Training via In-context Learning Theory
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models