Reading the Model's In-Context Task State as a Certified Safety Monitor

Aligned language models can be pushed into harmful behavior by the text placed before a request. A handful of fake "harmful question, compliant answer" examples is enough, and so is a slowly drifting conversation. Safety certificates are statements of the form "with 95% confidence, the harmful-output rate is below ε". Current certificates are computed for a fixed distribution of prompts, so they say nothing once the context changes. This proposal tests a specific idea. When a model learns a task from examples in its prompt (in-context learning), it compresses the inferred task into a single internal activation, a so-called task vector. If that is true, an attack that works by injecting harmful examples must move that activation. A monitor that reads the activation at the final token should therefore catch in-context attacks that text filters miss. Its alarm threshold can be set with conformal risk control, a calibration method that turns a small labeled holdout set into a finite-sample bound on error rates. We then ask the question that decides whether this matters in practice: does the bound stay valid under attacks the calibration set never saw, and does it survive the move from 7B to 70B models?

In plain terms, we build a cheap, training-light internal alarm for jailbreaks that arrive through context. We give it a statistical guarantee. We then check, at frontier scale, whether that guarantee actually holds when attackers change the context.

Core Claim

In-Context Learning Creates Task Vectors shows several things. Patching the hidden state from the last demonstration token into a query-only forward pass recovers 80–90% of few-shot accuracy. When the demonstrations conflict with the text, the patched vector rather than the text decides the output. Separately, Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations reports that ten harmful demonstrations raise Llama-2-7b-chat's HarmBench attack success rate from 3% to 50%.

Putting these together, we hypothesize that a successful in-context attack installs a "comply" task state. A refusal direction derived from a few demonstrations, read at the final prompt token, should then be a monitoring statistic that remains informative under shifts in the in-context attack. Concretely, let $h_L(x)$ be the residual-stream activation at layer $L$ and final token for full input $x$, which includes any conversation history. Let $v$ be the difference between mean task states induced by refusal demonstrations and by compliance demonstrations. The score is $s(x)=\cos(h_L(x), v)$, and the system intervenes when $s(x)<\tau$. The threshold $\tau$ is chosen by conformal risk control on $n$ labeled calibration inputs so that

$$\mathbb{E}\big[\mathbf{1}\{Y(x)=\text{harmful}\}\,\mathbf{1}\{s(x)\ge\tau\}\big]\le \alpha ,$$

where $Y(x)$ is the judged label of the model's sampled response and $\alpha$ is the user's budget for missed harmful outputs. A second calibration on benign inputs bounds the false-intervention rate. When the monitor fires, it either patches $v$ into the forward pass, which is a token-free version of in-context defense, or it refuses.

The guarantee holds only if test inputs are exchangeable with calibration inputs. The scientific content of the proposal is to measure how badly in-context and multi-turn shift break that assumption for an internal-state monitor, compared with text-level and trained monitors. We do this at 7B, 13B, and 70B.

Relation to Prior Work

Two papers motivate the gap. Both show in-context demonstrations steer safety but certify nothing. Using In-Context Learning to Improve Dialogue Safety retrieves prosocial demonstrations, which raises OPT-30B safety from about 60% to over 90%. Wei et al.'s in-context defense uses one or two refusal demonstrations to cut GCG success on Llama-2 from 21% to 0%. Neither reports a confidence bound, and neither asks whether the protection survives a stronger in-context adversary.

Two certification papers supply the statistical machinery. How Catastrophic is Your LLM? Certifying Risks in Conversation gives Clopper-Pearson bounds on multi-turn catastrophic risk over query-graph attack distributions. We reuse its attack distributions as shift tests, but it only evaluates; it does not defend. A Probabilistic Perspective on Unlearning and Alignment for Large Language Models shows that greedy-decoding evaluations hide leakage that appears in over 30% of sampled responses. This forces us to define $Y(x)$ over sampled outputs rather than greedy ones. That paper explicitly assumes static prompts, which is the assumption we stress.

The closest method is Selective Safety Steering via Value-Filtered Decoding. It trains a three-layer value head on hidden states and calibrates a token-rejection threshold with conformal risk control and Ville's inequality, on 7–8B models. We differ in three ways. First, our statistic comes from a few dozen demonstrations rather than a supervised head. Second, it is read once per prompt rather than at every token. Third, and most important, we test validity under in-context shift and at 70B, which that work does not.

Shh, don't say that! Domain Certification in LLMs obtains strong bounds but needs a second guide model, which is a cost our single-model read avoids. Certifying LLM Safety against Adversarial Prompting certifies against short token insertions. It cannot see fluent context manipulation, and its guarantees degrade with filter error. The same filter-error caveat applies to our judge, as discussed below.

Two lines of work constrain the design. GEOMETRIC SIGNATURES OF COMPOSITIONALITY ACROSS A LANGUAGE MODEL'S LIFETIME finds that meaning lives on a low-dimensional nonlinear manifold, with intrinsic dimension around 10, inside a linear subspace whose dimension grows with model width. Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders finds that steering behavior changes with scale and that wider dictionaries fragment concepts. Both are real reasons a single linear direction might fail at 70B. We therefore treat linearity at scale as a hypothesis to falsify, not an assumption. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal shows that robustness does not improve with parameter count. This is why scale is a stress axis here rather than a hope.

Research Program

The first stage is extraction and localization. For each model in Llama-2-7B-chat, Llama-2-13B-chat, Llama-3.1-8B-Instruct, and Llama-3.1-70B-Instruct, we build $v$ from 16–64 refusal and compliance demonstrations drawn from HarmBench-style behaviors that are disjoint from evaluation. We choose the layer by held-out separability, and we report how the best relative depth moves with scale. We also measure the local intrinsic dimension of task states around $v$, using the TwoNN estimator as in Lee et al. This lets us check whether a 70B failure coincides with the comply/refuse distinction becoming nonlinear.

The second stage is calibration. The harmful-risk calibration set mixes plain HarmBench and AdvBench prompts with in-context attacks at budgets $k\in\{0,2,5\}$. The benign set uses BeaverTails harmless questions and ProsocialDialog turns. Labels come from sampled responses scored by the HarmBench Llama-2-13B judge, which agrees with humans 93.19% of the time. We state certificates relative to this judge. On a 500-item subsample we re-label by hand and propagate judge error, so that we report a corrected bound alongside the nominal one.

The third stage is the shift battery. Its components were deliberately excluded from calibration: in-context attacks with $k\in\{10,20\}$, paraphrased demonstrations, C³LLM graph-path and adaptive-with-rejection conversations, and a white-box adaptive attacker who optimizes demonstrations to raise $s(x)$ while keeping the judged output harmful. For each condition we report the realized miss rate against the nominal $\alpha$, with Clopper-Pearson intervals.

The fourth stage measures cost: added latency and memory per query compared with value-filtered decoding and dual-model certification at 70B.

Baselines

The naive reference is the undefended model with a standard safety system prompt, evaluated under the same sampled-output protocol. This shows how much risk exists before any monitor is added.

The strongest training-free defense is Wei et al.'s in-context defense, which prepends one or two refusal demonstrations. It is the most direct competitor because it uses the same mechanism without a monitor. The reported number to beat is GCG success on Llama-2 reduced from 21% to 0%. We run it on identical prompts and sampled decoding, and attach a Clopper-Pearson bound so that both systems are compared on certified miss rate. It counts as beaten if, under the held-out attacks at $k\ge10$ and in multi-turn drift, our monitor's upper bound on miss rate is lower at a matched false-intervention rate.

The strongest certified competitor is value-filtered decoding, retrained per model on the same labeled pool and with the same calibration set and budget. No number is reported on Lacuna for value-filtered decoding's BeaverTails score, so we reproduce it rather than quote it. It counts as beaten if it violates its nominal $\alpha$ under shift more often than our monitor does, or if it matches validity only at higher inference cost.

Two ablations isolate the key ingredient. The first is a logistic probe trained at the same layer and token on the same labeled pool. It tests whether the demonstration-derived direction matters or whether any linear readout of that state works. The second is a text-level monitor: the HarmBench judge applied to the prompt plus history, conformally calibrated in the same way. It tests whether reading internal state matters at all.

All systems receive equal labeled data, the same calibration split, and the same hyperparameter search budget.

Why not just…?

Why not just certify the in-context defense with a binomial interval and stop? That certifies a static distribution and nothing else, which is exactly the limitation Scholten et al. name. It cannot adapt when an attacker adds more shots, and Wei et al. show attack strength grows quickly with the number of demonstrations. This alternative is a genuine threat. If in-context defense alone keeps the certified miss rate below $\alpha$ across the whole shift battery, the monitor is unnecessary, and we will report that outcome as the finding.

Why not just train a probe or value head on hidden states? With enough labels, a supervised readout will likely match or beat a demonstration-derived direction on the calibration distribution. Our bet is narrower: a direction tied to the causal task state should transfer better under unseen in-context shift. The probe ablation tests this directly. If the probe transfers equally well, the contribution reduces to the scale-and-shift validity study plus a cheaper per-prompt readout. We would say so and recommend the probe.

Why not just use a strong system prompt? Scholten et al. show that models which look clean under greedy decoding leak in over 30% of samples. Wei et al. show 10-shot in-context attacks overriding chat alignment, reaching 81% success on GPT-4 at 20 shots. Our naive baseline measures this directly under sampling.

Why not just filter text with erase-and-check? It certifies against short token insertions. Multi-turn drift of the kind Wang et al. document consists of individually benign turns that contain no removable adversarial substring. The text-monitor ablation measures this gap empirically rather than assuming it.

Decisive Checks

The idea succeeds only if three results hold together.

First, on the held-out shift battery, the internal-state monitor's realized miss rate stays at or below nominal $\alpha$ (within interval width), while the text-level monitor and value-filtered decoding exceed it in at least the $k\ge10$ and adaptive-conversation conditions.

Second, at 70B the direction still separates comply from refuse states, with held-out AUROC within a few points of 7B. If it fails, the failure should line up with rising local intrinsic dimension, which would confirm the nonlinearity concern from Lee et al. and point to a nonlinear readout.

Third, under the white-box adaptive attacker the guarantee degrades gracefully. We report the attacker budget at which the bound breaks.

Utility is checked as false-intervention rate on BeaverTails harmless prompts and ProsocialDialog, with PubMedQA accuracy as a capability sanity check. If the first result fails and in-context defense alone holds its bound, the proposal's central hypothesis is refuted.