TL;DR
This paper reframes activation steering—a technique for controlling LLM behavior—as a "budgeted allocation" problem, focusing on when to intervene rather than just how much. By introducing CLABS, a trajectory-aware gate based on classical feedback control, the authors demonstrate that selectively steering only the most "off-track" tokens achieves high behavioral control while significantly better preserving the model's original fluency and reasoning capabilities.
What Problem Does This Paper Address?
Activation steering is a popular, lightweight alternative to fine-tuning for controlling LLM behavior (e.g., making a model less toxic or more creative). It works by adding a specific "steering vector"—often derived from Sparse Autoencoders (SAEs)—to the model's internal hidden states during inference.
However, most current methods are "dense": they apply the steering signal to every single token generated. This is problematic because it pushes the model's internal representations away from the data distribution it was trained on. As the steering strength increases, the model's "fluency" erodes, leading to repetitive, incoherent, or low-quality text. The authors argue that many tokens in a sequence are already aligned with the target behavior; steering them adds no benefit but incurs a high "quality cost." The challenge is identifying the specific moments when the model is drifting off-course and intervening only then.
Methods
The authors propose a framework for sparse token-level steering, where a "gate" decides whether to steer each token subject to a fixed budget $p$ (the fraction of tokens allowed to be steered). They introduce two novel gating mechanisms:
- Contrastive Feature Projection (CFP): A memoryless gate that calculates the "alignment" of the current token with the target behavior using a dot product. It spends its budget on the tokens that are currently least aligned.
- Closed-Loop Adaptive Bernoulli Steering (CLABS): This is the paper's central contribution. It treats steering as a discrete-time feedback control problem. Instead of looking only at the current token, CLABS maintains a "leaky integral" (an exponentially weighted moving average) of past alignment errors.
- If the model has been consistently drifting away from the target behavior over several tokens, CLABS increases the probability of steering.
- This "memory" allows it to distinguish between a temporary dip in alignment and a sustained trajectory drift.
- The authors provide a theoretical stability analysis for CLABS, deriving a "gain region" that guarantees the controller won't oscillate or drift uncontrollably.
Both methods use directions derived from Sparse Autoencoders (SAEs), which decompose complex hidden states into interpretable, "monosemantic" features, allowing for more precise behavioral targeting.
Datasets
The researchers evaluated their methods across two open-weight models (Gemma 2 2B and Llama 3.1 8B) using three primary tasks:
- Toxicity Reduction: Using the RealToxicityPrompts dataset to suppress toxic continuations.
- Emotion Steering: Using the GoEmotions dataset to elicit specific affects like fear or sadness from neutral prompts.
- Instruction Following: Using AlpacaEval to ensure that steering for safety or emotion doesn't break the model's ability to follow general instructions.
Key Insights and Ideas
- Steering as Allocation: The most important variable in sparse steering isn't the magnitude of the push, but the selection of tokens. By fixing the budget and the vector, the authors show that "informed" gating (CFP/CLABS) vastly outperforms random or uncertainty-based gating.
- The Value of History: Memoryless gates (like CFP) can be "fooled" by momentary noise. CLABS’s use of an integral term allows it to focus its budget on sustained deviations, which is where steering is most effective.
- Efficiency: Neither gate requires an extra forward pass or expensive search (like Best-of-N). CFP adds just one inner product per token, and CLABS adds a few scalar operations—costs that are negligible ($10^{-6}$) compared to the model's standard forward pass.
- Stability Matters: By applying classical control theory to LLM internals, the authors move away from heuristic "trigger" rules toward provably stable intervention policies.
Key Results
- Superior Control at Low Budgets: As shown in Figure 1, CFP and CLABS achieve significantly larger behavioral shifts than other budgeted baselines (like entropy-based steering). CLABS is particularly dominant at very tight budgets (e.g., steering only 10-30% of tokens).
- Preserving Utility: In instruction-following tests (Table 1), CFP and CLABS were the only gates preferred over the unsteered model across all budgets. While "Full-steer" (steering 100% of tokens) caused a massive spike in perplexity (PPL) and a drop in diversity, CLABS maintained quality levels nearly identical to the base model.
- Efficiency Gain: At a moderate budget (50%), CLABS recovers 89–92% of the behavioral shift of "Full-steer" while avoiding almost all of the associated fluency degradation.
Limitations and Future Work
- SAE Dependence: The methods rely on having high-quality, pre-trained SAEs for the specific layers being steered.
- Single-Layer Focus: The study intervenes at a single mid-network layer. Future work could explore allocating a steering budget across multiple layers or multiple behavioral directions simultaneously.
- Linear Approximation: The stability proof relies on a first-order linear approximation of how steering affects alignment. While empirically successful, a full non-linear analysis of the decoding process remains an open challenge.
- Language Scope: The evaluation was limited to English text; cross-lingual steering behavior was not addressed.
Summary updated