Several recent safety systems for language models do not read the model's text output. They read its internal activations, the vectors of numbers the model computes at each layer, and use a small classifier called a probe to decide whether a request is harmful. These monitors are cheap and accurate on known attacks. Their weak point is an attacker who knows the probe and optimizes the prompt to push the probe's score down. The usual fix is to harden the probe with adversarial training or with a more complex nonlinear classifier. That fix tends to move the decision boundary on ordinary inputs, so the system starts refusing safe requests: the familiar trade-off between robustness and clean accuracy.
This proposal tests a different route. To hide harm from a probe and still get a harmful answer, an attacker has to drive the model's activations somewhere unusual. In the one case where evasion has been studied closely, the hidden activations collapsed into a much lower-dimensional subspace than normal. We therefore leave the accurate linear probe untouched and add a second check that fires only when the probe says "safe" and the activation also looks unlike anything seen in ordinary traffic. Benign inputs are, almost by definition, typical. The cost to clean accuracy is then capped by how often typical benign inputs get flagged as atypical, and that rate can be measured directly. The central experiment pits this two-part monitor against an attacker who knows both parts and optimizes against both at once.
Core Claim
Our claim is that, for activation monitors read at a fixed structural position, robustness to a white-box adaptive input attacker can be gained by constraining where an evasion can land, rather than by changing the probe. Concretely, we pair a linear harm probe with a detector of distributional atypicality fitted on benign and known-harmful activations. Against an attacker who has white-box access to both components, this pairing should give a lower attack success rate than an adversarially trained probe at the same benign false-positive rate. It should also force the successful evasions that remain to produce measurably weaker harmful content.
The claim is falsifiable. Suppose the attacker can cheaply find prompts whose activations are typical of benign traffic at the readout position yet still yield harmful completions. Then the model's internal state at that position carries no reliable harm signal under attack. That would be an informative negative result for the whole latent-monitoring line of work.
Grounding in Prior Work
Three papers motivate the premise that a harm signal survives attack. Jiachen Zhao et al. show that hidden states at the end of the instruction still encode harmfulness even when a jailbreak defeats the textual refusal. Maciej Chrabaszcz and Dhananjay Ashok et al. find that this signal appears during prefill, the forward pass over the prompt before any output token is produced, and that linear probes read it reliably. All three tested attackers who targeted the refusal, not the probe. We ask whether the signal survives once the probe itself becomes the target.
We take our host method from Any-Depth Alignment, or ADA. ADA re-inserts the chat template's assistant-header tokens partway through generation and reads a linear probe at that header. It keeps refusal near 100% under prefills of up to 2,500 tokens, cuts the attack success rate of GCG, AutoDAN, PAIR and TAP to below 3%, and keeps a near-0% false-positive rate on XSTest and on reasoning benchmarks. ADA was never tested against an attacker who knows the probe weights, and that is the gap we fill.
Several monitors from the same period share this blind spot, though each acknowledges it differently. PlugGuard and NExT-Guard monitor streaming generation with a learned dynamics head and with sparse-autoencoder features, respectively. GAVEL applies Boolean rules over 23 attention-derived "cognitive elements" and explicitly flags attacks that suppress those elements as an open risk. BASIS concedes that an attacker with access to its attention-ratio probes can find blind spots. We study the ADA-style readout because it is the strongest reported monitor on the shared jailbreak benchmarks. We also expect the geometric argument to carry over to these other monitors, and we test it on one of them.
Neural Chameleons both constrains and motivates the design. Models fine-tuned to evade probes drive deception-probe recall on Gemma-2-9B from about 90% down to 16–33%. That is a strong negative result for linear probes, and it explains why we do not simply trust the probe. The same paper also reports that the evasive activations are compressed: 11 principal components explain 90% of their variance, against 39 for normal activations. That compression is exactly the kind of atypicality our second detector is built to catch. Chameleons also notes that nonlinear probes resisted evasion better but paid for it in false positives. This is the robustness-accuracy trade-off, now inside the monitor, and our design is meant to avoid it.
Chameleons differs from our setting in two ways. Its evasion is installed by fine-tuning and switched on by an explicit trigger phrase. Our attacker cannot touch the weights and must reach evasion through the prompt alone. Chameleons also proposes adversarial training of monitors but does not implement it, so we implement it here as our main baseline.
Research Program
The monitor reads hidden states at the injected assistant-header position at two or three layers between 50% and 75% of model depth. We choose this band following PlugGuard's observation that mid-to-late layers carry risk information. At each layer it computes two quantities.
The first is a linear harm probe, trained exactly as in ADA. Its output is $p_\ell(h)$, the probe's harm score for the hidden state $h$ at layer $\ell$.
The second is an atypicality score: the distance from $h$ to the nearer of two class-conditional Gaussians, one fitted on benign activations and one on known-harmful activations,
$$a_\ell(h) = \min_{c \in \{\text{benign},\,\text{harmful}\}} (h-\mu_{c,\ell})^\top \Sigma_{c,\ell}^{-1} (h-\mu_{c,\ell}).$$
Here $\mu_{c,\ell}$ and $\Sigma_{c,\ell}$ are the mean and a shrinkage-regularized covariance of class $c$ at layer $\ell$. A state is flagged if any layer's probe fires, or if the states are atypical at every monitored layer. Thresholds are set so that the combined false-positive rate on a held-out benign split is 1%. The detector is deliberately simple: a skeptic should not be able to attribute any gain to classifier capacity.
The attacker knows everything: model weights, probe weights, Gaussian parameters, and thresholds. It optimizes a prompt suffix to minimize
$$\mathcal{L} = \mathcal{L}_{\text{target}} + \lambda_p \sum_\ell \sigma\big(p_\ell(h)\big) + \lambda_a \sum_\ell \mathrm{softplus}\big(a_\ell(h) - \tau_\ell\big).$$
In this objective, $\mathcal{L}_{\text{target}}$ is the usual loss for producing a harmful target completion, $\sigma$ is the logistic function, $\tau_\ell$ is the atypicality threshold at layer $\ell$, and $\lambda_p$ and $\lambda_a$ are swept weights. We implement the attack in three forms. The first is GCG-style discrete token search. The second is a fluency-constrained variant, motivated by AutoDAN, which shows that fluent attacks bypass surface filters. The third is a continuous embedding-space attack in the spirit of LARGO. The embedding attack is stronger than any real input attacker, so it serves as an upper bound. Prefill attacks are included because ADA's main claim concerns them.
The analysis has three parts. First, we trace the attacker's Pareto frontier: harmful-output success against total detection score, as $\lambda_p$ and $\lambda_a$ vary. Second, we measure how harmful the successful evasions actually are, using StrongREJECT-style graded scoring rather than a binary success label. Third, we test whether evasions that do succeed occupy a compressed subspace, as Chameleons found. We do this by computing the participation ratio of activations across many evasive prompts.
Baselines
The do-nothing reference is the aligned model with no monitor, attacked by the same optimizer. It sets the ceiling on attack success.
The strongest existing method is ADA's linear probe as published. On the benchmarks we reuse, Any-Depth Alignment reports attack success below 3% against GCG, AutoDAN, PAIR and TAP, and a near-0% false-positive rate on XSTest. We reproduce it on the same models and probe-training data. We first confirm that we recover these numbers against non-adaptive attacks. We then attack it with the probe-aware loss, setting $\lambda_a = 0$. We expect this probe-only adaptive attack to succeed often. If it does not, the premise of the proposal is weaker than we think, and we will report that.
The most dangerous competitor is an adversarially trained probe: the defense proposed but not built in Neural Chameleons. We build it by alternating attack generation with probe retraining, as a latent-space analogue of the dynamic attack pool in R2D2 from HarmBench. We test it in two forms, linear and two-layer MLP. Fairness rests on two controls. Its total attack-generation budget, counted in optimizer steps, matches the budget used to fit our detector plus a calibration round. And every defense is compared at the same 1% benign false-positive rate.
Three ablations isolate the key ingredient. A multi-layer probe ensemble without the atypicality term tests whether gains come merely from monitoring several layers. An atypicality-only detector tests whether the probe is needed at all. A single-layer version of our monitor tests whether requiring atypicality at every monitored layer matters.
Two external reference points come from outside the monitor family. The first is the perplexity filter from Baseline Defenses, which catches nearly all GCG suffixes at roughly a 10% benign false-positive rate. The second is an external text guardrail. We found no reported result on Lacuna for Llama Guard's attack success rate on these benchmarks, so we will measure it ourselves.
We beat the field if two conditions hold under the fully adaptive attacker at 1% benign false-positive rate. Our monitor's attack success must be clearly below both the probe-only adaptive number and the adversarially trained probe; we set the bar at a halving of the adversarially trained probe's success rate. And its benign utility must match ADA's.
Why not just…?
Why not just adversarially train the probe? This is the real threat. If an adversarially trained linear probe reaches equal robustness at equal false-positive rate, our second detector adds nothing, and we will say so. Two pieces of evidence suggest it will fall short. Chameleons found that more robust nonlinear probes cost false positives. And adversarial training only covers the evasions its attack pool happened to find, which HarmBench notes is the general limitation of R2D2. Our atypicality detector, by contrast, is fitted without any attack data. The head-to-head comparison at matched false-positive rate and matched attack-generation budget decides the question. We also test generalization by training the hardened probe against GCG-style attacks and evaluating against the embedding-space and fluent attacks it never saw.
Why not just filter inputs by perplexity or paraphrase them? Baseline Defenses reports roughly a 10% benign false-positive rate for perplexity filtering, which is ten times our operating point. Fluent attackers such as AutoDAN evade it. Neither filtering nor paraphrasing addresses prefill attacks, where the attacker writes the start of the assistant's reply. We run the perplexity filter against our fluency-constrained attacker to confirm this directly.
Why not just use an external guardrail model on the text? GAVEL reports GPT-4's detection recall falling as low as 12% in some misdirection categories. ADA reports about 500ms of latency and nearly 1GB of memory for an external guard on a 10,000-token context, against about 25ms and 2–3MB for the probe. An external guard is also a fixed boundary that an adaptive attacker can target, as the Black-Box Guardrail Reverse-engineering Attack shows by extracting commercial guardrails with rule-matching rates above 0.92. We include the guard as a measured baseline rather than dismissing it.
Proposed Experiments
We use Llama-3-8B-Instruct, Gemma-2-9B-it and a Qwen instruct model, all of which have the structural header tokens ADA requires. Harmful behaviors come from AdvBench, JailbreakBench, HarmBench and StrongREJECT. Attack success is judged by the HarmBench classifier, which agrees with human labels 93.19% of the time, using HarmBench's fixed 512-token completion length; HarmBench shows success rates can shift by up to 30% when that length changes.
To measure the cost to benign users, we use the XSTest safe prompts, paying special attention to its safe-context and fiction categories, where Llama 2 refused 24 of 25 prompts and GPT-4 refused over 50% respectively. We also use JailbreakBench's 100 benign counterparts. For utility, we run GSM8K, MATH, MMLU and GPQA Diamond with the monitor active on long chain-of-thought generations, and count any interruption as a failure. This addresses the untested worry that long, legitimate reasoning drifts into atypical or risky-looking regions.
As a secondary stress test, we take a Chameleons-style fine-tuned model organism and ask whether our detector catches its trigger-induced compression without being retrained. We also transfer the atypicality component to PlugGuard's streaming head and test it on StreamGuardBench.
The decisive checks are these. Under the fully adaptive attacker, does our monitor's attack success at 1% benign false-positive rate fall well below that of the adversarially trained probe? Do the attacks that still succeed produce lower graded harm scores, which would mean evasion and harm compete? Does the benign false-positive rate on long reasoning and on XSTest's gray-area categories stay at the calibrated level?
The proposal fails if the embedding-space attacker reaches high success using activations that stay typical at every layer. That outcome would show that hard-to-reach harmful states are not a defensive asset, and that latent monitoring needs a different basis.
References
- LLMs Encode Harmfulness and Refusal Separately
- Efficient Large Language Models Moderation with Multi-Layer Latent Prototypes
- Language Models Can Predict Their Own Behavior
- Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
- PlugGuard: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detection
- NExT-Guard: Training-Free Streaming Safeguard without Token-Level Labels
- GAVEL: Towards Rule-Based Safety through Activation Monitoring
- BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes
- Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors
- AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models
- LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Black-Box Guardrail Reverse-engineering Attack