The Rogue Scalpel: Activation Steering Compromises LLM Safety

Anton Korznikov, Andrey V. Galichin, Alexey Dontsov, Oleg Y. Rogov, Ivan V. Oseledets, Elena Tutubalina

Abstract: Activation steering is a promising technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference. It is often framed as a precise, interpretable, and potentially safer alternative to fine-tuning. We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests. Through extensive experiments on different model families, we show that even steering in a random direction can increase the probability of harmful compliance from 0% to 2-27%. Alarmingly, steering benign features from a sparse autoencoder (SAE), a common source of interpretable directions, increases these rates by a further 2-4%. Finally, we show that combining 20 randomly sampled vectors that jailbreak a single prompt creates a universal attack, significantly increasing harmful compliance on unseen requests. These results challenge the paradigm of safety through interpretability, showing that precise control over model internals does not guarantee precise control over model behavior.

Introduction

Large Language Models (LLMs) are typically secured through alignment processes like Reinforcement Learning from Human Feedback (RLHF), which train the model to refuse harmful prompts. However, this alignment is often viewed as a "black box," where the internal mechanics of refusal remain opaque.

Activation steering has emerged from the field of mechanistic interpretability as a potential solution. By injecting human-interpretable "directions" (vectors) into a model's latent space during inference, researchers hoped to achieve surgical control over model behavior without the need for computationally expensive fine-tuning. This paper demonstrates that this "surgical" tool is actually a "rogue scalpel." Rather than providing safe control, activation steering systematically compromises safety guardrails, often through seemingly benign or even random perturbations.

Conceptual overview of activation steering: injecting a concept vector (e.g., "Portuguese") can inadvertently bypass safety filters, leading to harmful compliance.

Methodology

The researchers evaluated the safety impact of activation steering across several model families, including Llama-3.1, Qwen2.5, and Falcon3. The core approach involved Activation Injection: adding a steering vector ($v$) scaled by a coefficient ($\alpha$) to the model's residual stream at specific layers.

Two primary sources for these vectors were tested:

  1. Random Directions: Vectors sampled uniformly from a unit sphere, serving as a baseline for stochastic noise.
  2. SAE-based Directions: Features extracted from Sparse Autoencoders (SAEs), which are currently the gold standard for identifying interpretable concepts within a model's internal representations.

To evaluate safety, the team utilized JailbreakBench, a standard benchmark of 100 harmful prompts. Responses were classified as SAFE or UNSAFE using an LLM-as-a-judge (Qwen3-8B) based on established jailbreak criteria. Additionally, the MMLU benchmark was used to ensure that steering wasn't simply "breaking" the model's general intelligence while bypassing safety.

Key Insights: The Vulnerability of Internal Circuits

Interpretability is Not a Safety Guarantee

Perhaps the most significant finding is that steering with "benign" SAE features—such as those representing "brand identity" or "modal verbs"—was just as effective at inducing harmful compliance as random noise. This suggests that a feature's semantic meaning to a human auditor does not reflect its functional impact on the model's safety circuits.

The "Sweet Spot" for Sabotage

The experiments revealed a specific range of steering strength ($\alpha$) where safety mechanisms fail, but general reasoning capabilities remain intact. As shown in the data, there is a threshold where compliance spikes while the model continues to produce coherent, albeit harmful, text.

Harmful compliance rates as a function of steering strength across different layers of Llama-3.1-8B, showing the sensitivity of middle layers.

Middle Layers as the Weak Point

Steering is most effective when applied to the middle layers of the transformer architecture. This indicates that safety mechanisms are not hard-coded at the input or output stages but are emergent properties of intermediate processing. These intermediate stages are highly vulnerable to internal perturbations.

Experimental Results

  • Random Steering Efficacy: Simply adding random noise to activations increased harmful compliance from 0% to as high as 13%.
  • SAE Feature Danger: Steering with SAE features yielded a 1–4% higher compliance rate than random noise, even when the features were semantically harmless. In total, 817 out of 1,000 tested SAE features were able to jailbreak at least one harmful prompt.
  • Universal Attacks: By averaging just 20 "lucky" random vectors that successfully jailbroke a single prompt, the authors created a universal attack vector. For Llama-3.1-70B, this simple method achieved a 50.4% success rate in jailbreaking the model across all harm categories.

Conclusion and Future Work

The findings suggest a fundamental "mitigation gap" in current LLM development. Safety training that focuses on input-output pairs does not account for activation-level vulnerabilities. The authors argue that future adversarial training must include steering-based perturbations to build truly robust models. While we now know that steering breaks safety, the exact circuit-level reason why a benign vector can disable a refusal mechanism remains a critical mystery for future mechanistic interpretability research.

Summary updated

Abstract

Activation steering is a promising technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference. It is often framed as a precise, interpretable, and potentially safer alternative to fine-tuning. We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests. Through extensive experiments on different model families, we show that even steering in a random direction can increase the probability of harmful compliance from 0% to 2-27%. Alarmingly, steering benign features from a sparse autoencoder (SAE), a common source of interpretable directions, increases these rates by a further 2-4%. Finally, we show that combining 20 randomly sampled vectors that jailbreak a single prompt creates a universal attack, significantly increasing harmful compliance on unseen requests. These results challenge the paradigm of safety through interpretability, showing that precise control over model internals does not guarantee precise control over model behavior.
BibTeX
@misc{korznikov2025rogue,
  author = {Anton Korznikov and Andrey V. Galichin and Alexey Dontsov and Oleg Y. Rogov and Ivan V. Oseledets and Elena Tutubalina},
  title = {{The Rogue Scalpel: Activation Steering Compromises LLM Safety}},
  year = {2025},
  eprint = {2509.22067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22067},
}
Figure