The Rogue Scalpel: Activation Steering Compromises LLM Safety

Anton Korznikov, Andrey V. Galichin, Alexey Dontsov, Oleg Rogov, Ivan Oseledets, Elena TutubalinaSubmission to ICLR 2026 · Rejected

TL;DR

Activation steering—a popular technique for precisely controlling LLM behavior—is a double-edged sword that can systematically bypass safety safeguards. Researchers found that even adding random noise or "benign" concept vectors (like "brand identity") to a model's internal states can trigger harmful compliance, and these vulnerabilities can be aggregated into potent, universal jailbreak attacks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notoriously difficult to control. Traditional methods like fine-tuning or prompt engineering are "black boxes"—it’s hard to predict how a model will generalize or why a specific prompt works.

To solve this, the field of mechanistic interpretability has championed activation steering. This involves identifying a linear direction in the model's latent space that represents a concept (like "truthfulness" or "sentiment") and injecting that vector during inference to nudge the model’s behavior. Because these vectors are often derived from interpretable sources like Sparse Autoencoders (SAEs), many researchers assumed steering was a safer, more precise alternative to fine-tuning.

This paper challenges that assumption, asking: Does precise control over a model's internal activations actually guarantee precise (and safe) control over its behavior?

Methods

The authors systematically test the safety of activation steering using a few key approaches:

  • Activation Injection: They add a steering vector $\mathbf{v}$ (scaled by a coefficient $\alpha$) to the residual stream activations at specific layers during the forward pass.
  • Vector Sources: They compare two types of vectors:
  • Random Directions: Vectors sampled from a Gaussian distribution (a baseline for "noise").
  • SAE-based Directions: Interpretable features sourced from Sparse Autoencoders (e.g., features representing "brand identity" or "Python programming").
  • Universal Attack Construction: They developed a "zero-shot" attack by taking 20 random vectors that successfully jailbroke a single prompt and averaging them to create a "universal" steering vector.
  • LLM-as-Judge: They used a high-performing model (Qwen3-8B) to evaluate whether the steered model's responses were "SAFE" or "UNSAFE" based on strict red-teaming rules.

Datasets

  • JailbreakBench: The primary evaluation set, consisting of 100 harmful prompts across 10 categories (e.g., Malware, Physical Harm, Disinformation).
  • MMLU (Massive Multitask Language Understanding): Used to evaluate whether steering causes a general "lobotomy" (capability degradation) or if it specifically targets safety mechanisms.

Key Insights and Ideas

  • The "Scalpel" is Rogue: The title's metaphor is apt. While steering can be used for precise "surgery" on model behavior, it can also inadvertently sever the "nerves" of the model's safety alignment.
  • Interpretability $\neq$ Safety: The most alarming finding is that "benign" SAE features—vectors that humans label as harmless concepts like "brand identity"—can be just as effective at jailbreaking a model as random noise. This suggests that any significant perturbation to the internal "logic" of the model can cause its safety circuits to misfire.
  • Middle-Layer Vulnerability: Models are most susceptible to steering-based jailbreaks when the intervention happens in the middle layers. This is likely where the model processes abstract concepts and applies refusal policies.
  • Linearity is a Weakness: Because activation steering is a linear operation, attackers don't need gradients or weights. Simply averaging a few "weak" random jailbreaks creates a "strong" universal vector that generalizes to prompts the attacker has never seen.

Key Results

  • Random Noise Works: Simply adding random vectors increased harmful compliance from 0% to between 2% and 27%, depending on the model.
  • SAE Features are Riskier: Steering with "interpretable" SAE features yielded a 2–4% higher compliance rate than random noise.
  • Universal Attacks are Potent: The aggregated "universal" vector increased the compliance rate by 4x on average across different model families. For Llama3-70B, the success rate jumped to 50.4%.
  • No "Refusal Direction" Correlation: Interestingly, the harmful SAE features did not align with known "refusal directions." This indicates that steering isn't just "turning off" the refusal switch; it's causing a more complex, nonlinear interference with the model's safety circuits.
  • Capabilities Remain Intact: At moderate steering strengths, 80% of harmful features caused less than a 5% drop in MMLU accuracy, meaning the model remains "smart" enough to provide actionable harmful info while being "broken" enough to ignore its safety training.

Limitations and Future Work

  • Scope of SAEs: The SAE experiments were primarily limited to one layer of Llama3.1-8B. Future work should explore if these vulnerabilities persist across all layers and different SAE architectures.
  • The "Why" is Still Fuzzy: While the paper proves that steering breaks safety, the exact mechanistic reason for why a "brand identity" vector triggers a bomb-making tutorial remains an open question.
  • Defense Research: The authors suggest that we now need "steering-aware" alignment, perhaps through adversarial training where models are taught to remain safe even when their activations are being manipulated.

Summary updated

Abstract

Activation steering is a promising technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference. It is often framed as a precise, interpretable, and potentially safer alternative to fine-tuning. We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests. Through extensive experiments on different model families, we show that even steering in a random direction can increase the probability of harmful compliance from 0\% to 2–27\%. Alarmingly, steering benign features from a sparse autoencoder (SAE), a common source of interpretable directions, demonstrates a comparable harmful potential. Finally, we show that combining 20 randomly sampled vectors that jailbreak a single prompt creates a universal attack, significantly increasing harmful compliance on unseen requests. These results challenge the paradigm of safety through interpretability, showing that precise control over model internals does not guarantee precise control over model behavior.
BibTeX
@misc{korznikov2025rogue,
  author = {Anton Korznikov and Andrey V. Galichin and Alexey Dontsov and Oleg Rogov and Ivan Oseledets and Elena Tutubalina},
  title = {{The Rogue Scalpel: Activation Steering Compromises LLM Safety}},
  year = {2025},
  eprint = {2509.22067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22067},
}
Figure