Lazy Activation Filters: Inference-Time Safety Whose Cost Does Not Grow With Its Data

Most ways of making a language model safer change how it behaves on every input, harmful or not. The cost to everyday ability is called the alignment tax: lost accuracy on math, code and general questions, plus refusals of harmless requests. In robotics, the same safety-versus-performance conflict is usually handled with a predictive safety filter. This is a small optimizer that predicts a few steps ahead, does nothing while the predicted trajectory stays safe, and applies the smallest possible correction only when it foresees a violation. We propose building such a filter inside a language model's hidden activations. It would look ahead across a block of transformer layers and solve a tiny constrained problem only when the activation trajectory is predicted to enter a harmful region. On every other token it applies exactly zero intervention.

The idea depends on two lessons from data-driven control. First, a filter defined by recorded data has an alignment tax set mostly by coverage. Benign inputs unlike anything in the "benign" reference data get distorted. The only fix is more data, so the filter's optimization must not grow with the amount of data. Control theory has a recipe for this: parameterize the problem by running second-moment statistics rather than raw samples. Second, the filter predicts with a linearization of a nonlinear network. It therefore has to know how far that linearization can be trusted, especially under adversarial prompts. The core claim is that a filter with three properties can drive the tax on non-triggered inputs to zero by construction while matching current safety levels. Those properties are laziness, fixed problem size as the benign library grows, and safety margins sized from a measured linearity radius. The tax that remains on triggered inputs can then be measured and attributed.

What Exists and What Is Missing

Methods that motivate the proposal. Two prior methods supply most of what this proposal builds on.

  • Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control shows that layer-to-layer propagation is close to linear-time-varying. It uses this to steer with a closed-loop linear-quadratic regulator (A-LQR). A-LQR reports a 30–50x toxicity reduction on RealToxicityPrompts, against 8–10x for open-loop ITI and ActAdd. We reuse its Jacobian-based layer model. A-LQR has no hard constraints, though, and it steers continually rather than only on demand. The same paper shows an adaptive variant of its own method (A-LQR+) achieving near 100% refusal bypass. That result constrains us: any internal filter must be evaluated against attacks that know the filter.
  • AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint learns a linear map that sends benign activations to approximately zero steering. It reports over 90% defense success across GCG, AutoDAN, PAIR and ReNeLLM. On AlpacaEval it scores 67.3% utility against 67.1% for vanilla Llama-3.1-8B. Its "approximately zero" is a soft regression fit on a static benign set, applied one layer at a time without lookahead. A benign query outside that set's span is steered anyway.

Methods that set the safety bar. Training-time projection methods are the main reference for safety and tax.

Both name the same weaknesses: a static probe set, sensitivity to the singular-value threshold, and decomposition cost that grows with hidden size.

Methods that supply the control machinery. The filter's structure comes from robotics: Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning and Differentiable Predictive Control for Robotics: A Data-Driven Predictive Safety Filter Approach. The data-volume fix comes from two lines of work:

None of these has been applied inside a foundation model. Their robotic settings have tens of state dimensions, not 4096.

Research Program

The filter. We pick a block of $H$ consecutive layers starting at layer $\ell$ and model it with the time-varying linearization $h_{k+1} = A_k h_k + B_k u_k$. Here $h_k$ is the residual-stream activation at layer $k$ and $u_k$ is an additive correction. As in A-LQR, $A_k$ and $B_k$ are averaged Jacobians.

The harmful region is a polytope $\{h : W h > b\}$ with a fixed number $F$ of facets. Each row of $W$ is a linear harmfulness probe fit to a cluster of the harmful-activation library. At each generated token, the filter predicts the uncorrected trajectory. If every predicted layer satisfies $W h_k \le b - \rho_k$, it returns $u=0$. Otherwise it solves

$$\min_{u_\ell,\dots,u_{\ell+H-1}} \sum_k u_k^\top \left(R + \lambda\, \hat{\Sigma}_{\text{benign}}\right) u_k \quad \text{s.t.} \quad W \hat h_k(u) \le b - \rho_k,\; k=\ell+1,\dots,\ell+H.$$

The symbols are as follows:

  • $\hat h_k(u)$ is the linearly predicted activation under the corrections.
  • $R$ is a small identity weight.
  • $\hat{\Sigma}_{\text{benign}}$ is the running second moment of benign activations.
  • $\lambda$ sets how strongly corrections avoid directions that benign inputs use.
  • $\rho_k$ is a safety margin.

Why the cost stays fixed. The benign library enters only through $\hat{\Sigma}_{\text{benign}}$, and the harmful library only through $W$ and $b$. The problem therefore has $Hd$ variables and $HF$ constraints whether the libraries hold a thousand or a hundred million token activations. This is the covariance-parameterization idea carried over.

To keep the $d \times d$ matrix tractable, we maintain a rank-$r$ sketch plus an isotropic residual, updated incrementally. The active constraint set is usually one or two facets. The resulting quadratic program is then a small dual problem with closed-form solves per facet. We expect the cost on triggered tokens to be comparable to A-LQR's reported 10–20 tokens/s slowdown on an RTX 4090. Non-triggered tokens cost one linear prediction.

Why soft weighting instead of a hard null space. A hard null space shrinks to nothing as benign data accumulates, because the benign second moment becomes full rank. This is exactly the threshold-sensitivity problem NSPO and GuardSpace report. A weighted quadratic penalty degrades gracefully instead. The regularization analysis in the covariance-parameterization paper guides how $\lambda$ should scale with record length.

Margins from measured nonlinearity. The linear prediction is only trustworthy near the linearization point. As a standalone study, we will measure the validity radius on Llama-3-8B and Llama-3.1-8B. This is the perturbation size $\|\delta\|$ at which the true block output differs from $A\delta$ by more than a fraction $\epsilon$ of $\|A\delta\|$. We measure it separately for benign prompts, plain harmful prompts, and prompts carrying GCG and AutoDAN suffixes.

The margin $\rho_k$ is set from the empirical linearization-error quantile at the required correction size. This is the language-model analogue of a robust tube. If the required correction exceeds the validity radius, the filter does not trust its prediction. It applies the correction, runs the true layers, re-checks the constraint, and iterates up to three times, like a truncated sequential quadratic program. If the constraint still fails, it falls back to refusal. Fallback frequency becomes a reported quantity.

The radius study answers on its own an open question the file identifies: does local linearity survive adversarial inputs? A sharp collapse of the radius under jailbreak suffixes would itself be a useful negative result. It would bound every linear-steering method, A-LQR and AlphaSteer included.

Growing the libraries. Benign activations stream into $\hat{\Sigma}_{\text{benign}}$ continuously. Harmful facets are refit only when new harmful samples are selected for their effect on constraint feasibility, following the online-selection logic in the constraint-guided data-selection paper. This keeps $F$ fixed.

Baselines

Reference points. The naive reference is the unmodified instruction-tuned model, which sets both the zero-tax utility ceiling and the unprotected attack rate. A second reference is the external-classifier route analyzed in Control Tax: The Price of Keeping AI in Check. That paper reports that a 15% safety gain from switching monitors cost a 45x increase in cost. We include a Llama-Guard-style output classifier at matched latency, to place the filter on the same cost–safety curve.

AlphaSteer, the strongest direct competitor. It is inference-time, null-space-motivated, and reports essentially zero utility loss on Llama-3.1-8B (67.3% vs 67.1% AlpacaEval) with over 90% defense success. We run the released recipe on the same benign and harmful libraries, the same model, and the same layers. We also run a streaming variant that refits its map from the same running second moment our filter uses. This removes "more data" as an explanation for any gap. Beating it means matching its defense success within two points on each attack. It also means showing lower degradation on benign tasks outside the library's coverage, specifically MATH and code when the library was built from general instructions only.

A-LQR. It is the strongest closed-loop method. It is evaluated on RealToxicityPrompts and TruthfulQA as in its paper, and on our refusal suites with its setpoint retargeted to refusal.

Probe-gated A-LQR. This turns A-LQR on only when a single-layer harmfulness probe fires. It is the simplest way to get laziness without our machinery, so it is the key competitor for the claim that lookahead and margins matter.

Ablations. Four ablations isolate the ingredients:

  • removing the horizon ($H=1$);
  • removing the margins ($\rho=0$);
  • replacing the soft benign penalty with a hard truncated null space at several thresholds;
  • freezing the benign library at its initial size.

All methods share the tuning budget: the same number of hyperparameter configurations, selected on a held-out validation split of HarmBench and AlpacaEval.

Training-time reference. NSPO's 0.18% HarmBench attack success is cited as a reference point, not a matched baseline, because it changes weights with GRPO training. If compute allows, we report it under identical evaluation.

Why not just…?

Why not just gate A-LQR with a probe? This is the real threat. If it matches the filter on every axis, the honest conclusion is that laziness, not prediction, carries the benefit. We expect the gate to fail in two specific places.

  • Delayed harm. In multi-turn and ReNeLLM-style nested attacks, the harmful direction emerges deeper in the network or later in the generation. A single-layer gate misses these, while a horizon over layers sees them coming.
  • Over-steering. A-LQR tracks a fixed setpoint rather than solving for the smallest correction that meets the constraint, so it overshoots once triggered.

The $H=1$ ablation and the gated baseline settle this directly. We will report both regardless of outcome.

Why not just give AlphaSteer more benign data? If coverage is the problem, refitting AlphaSteer on a larger library might suffice, and its map can also be fit from second moments. That is why the streaming AlphaSteer variant exists. The remaining differences are threefold. Our intervention is exactly zero rather than regression-approximately zero on non-triggered tokens. Its size is chosen per token rather than fixed. And it accounts for layer coupling. A-LQR's own comparison shows open-loop steering plateauing at 8–10x where closed-loop control reaches 30–50x. That suggests coupling matters, but it does not prove the point for refusal, so the experiment does.

Why not just train safety in with NSPO? Training-time projection already reports small tax and very low attack success. It fixes one probe set at training time, though. A deployed model meets new capability domains, and NSPO would need retraining and a fresh decomposition each time. The filter only updates a running sketch. If NSPO's tax on out-of-coverage domains is as small as on in-coverage ones, the inference-time argument weakens to one of flexibility alone. The streaming experiment below tests this.

Proposed Experiments

Models and benchmarks. We use Llama-3.1-8B-Instruct and Llama-3-8B-Instruct, matching the AlphaSteer and A-LQR settings. Safety is measured with HarmBench and JailbreakBench attack success, and with AdvBench defense success under GCG, AutoDAN, PAIR and ReNeLLM. Over-refusal is measured with XSTest. Utility is measured with AlpacaEval, MATH, GSM8K and MMLU.

Coverage stream. The benign library starts from general instruction data and grows from roughly $10^3$ to $10^7$ token activations by streaming in math, code and multilingual text. At each size, we measure three things: utility on held-out MATH, GSM8K and HumanEval-style code; XSTest over-refusal; and per-token filter latency. The prediction is that utility loss on not-yet-covered domains falls as those domains enter the library, while latency stays flat. AlphaSteer refit from raw samples should show growing fit time.

Decisive checks.

  • Laziness on benign inputs. The first check is the fraction of benign tokens receiving any nonzero correction. If this is not below a few percent on AlpacaEval and MATH, the laziness premise fails.
  • Where the tax lives. On benign prompts that do trigger the filter, we compare utility to the unmodified model. This tells us whether the residual tax comes from false triggers or from correction size.
  • Adaptive attacks. We run an adaptive attack in the spirit of A-LQR+ with full knowledge of the filter's probes and Jacobians. Defense success under this attack is the headline robustness number. A collapse here, given A-LQR's own near-100% bypass result, would be reported as the main limitation.
  • Validity radius. The radius study is judged by whether its margins predict failures. Filter failures should concentrate on inputs whose required correction exceeded the measured validity radius. If failures are instead uniform, the margin mechanism is not doing its job and should be dropped.