Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions

Diancheng Kang, Zheyuan Liu, Ningshan Ma, Yue Huang, Zhaoxuan Tan, Meng JiangarXiv 2026 · arXiv

TL;DR

Activation steering—controlling LLM behavior by injecting vectors into internal representations—often causes models to become incoherent during long, multi-turn conversations. This paper introduces Gated Cropped Attention-Delta steering (GCAD), which prevents this "coherence collapse" by intervening at the attention level rather than the residual stream and using selective gating to mimic how models naturally process prompts.

What Problem Does This Paper Address?

Activation steering is a popular "middle ground" between brittle prompting and expensive fine-tuning. By adding a "persona vector" to a model's internal activations, researchers can steer it toward specific traits like honesty or a particular persona.

However, the authors identify a critical failure mode: KV-cache contamination. In a stateful, multi-turn dialogue, the model stores its previous (steered) states in its Key-Value (KV) cache. When generating new tokens, the model re-attends to these perturbed states. Standard steering methods inject the same perturbation at every single decoding step, leading to a cumulative "error" that eventually causes the model to lose the plot, repeating itself or becoming entirely nonsensical.

Figure 1: Comparison of standard persona-vector steering, prompt-only control, and a no-steering baseline across dialogue turns, showing cumulative degradation in response coherence under standard steering together with the corresponding trait-expression trend.

Methods

The authors propose GCAD, which moves away from the standard practice of uniform residual-stream injection. Their approach is built on three pillars:

  1. Attention-Level Intervention (P1): Instead of adding vectors to the post-MLP residual stream, GCAD targets the output of the self-attention mechanism. The authors argue that since prompts naturally influence behavior through attention, steering interventions should follow the same pathway.
  2. Cropping (P2): To avoid the feedback loop of KV-cache contamination, GCAD extracts steering signals only from the system-prompt's contribution to the attention output. It ignores the contributions from previously generated (and already steered) response tokens.
  3. Token-Level Gating (P3): Natural prompting doesn't affect every token equally; only specific "trait-bearing" words show strong alignment with a persona. GCAD uses a sigmoid gate to apply steering strength dynamically. It calculates a "compatibility score" between the current token's query and the system prompt's keys, steering heavily only when the model is actively engaging with the prompt's context.
Figure 3: Steering-signal extraction for GCAD. Standard persona steering forms a residual-stream vector by differencing post-MLP hidden states under contrastive system prompts. GCAD instead extracts per-layer attention-delta vectors from the self-attention output, then crops the attention sum to retain only system-prompt source-token contributions (Eq. 7). The resulting cropped attention-delta vectors are applied at inference with the per-token gate in Eq. 10.

Datasets

The researchers evaluated their method using two primary models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct.

They tested 15 persona traits across three categories:

  • RLHF-aligned: (e.g., Honest, Polite)
  • RLHF-opposing: (e.g., Evil, Sycophantic)
  • Neutral: (e.g., Creative, Humorous)

The evaluation used a 10-turn dialogue benchmark consisting of 60 conversations per trait, with GPT-4o-mini serving as the judge for both trait expression and response coherence.

Key Insights and Ideas

  • Prompt-Activation Duality: The core philosophy is that steering should "act like a prompt." If a prompt works by changing how tokens attend to each other, steering should intervene at that specific site.
  • Uniform Steering is "Unnatural": The authors show that standard steering pushes every token toward the target direction, whereas a system prompt only affects a sparse subset of tokens. This "over-steering" is what kills coherence.
  • The Attention Pathway is the Signal, the MLP is the Noise: By decomposing the residual stream, the authors found that the attention mechanism carries the vast majority of the "trait signal" induced by a prompt. Injecting into the MLP pathway or the raw embedding difference provides much weaker control.
Figure 2: Per-token cosine similarity with the persona vector under activation steering (left) and system prompting (right), using the same question and the evil trait. Color denotes cosine similarity with the persona direction. System prompting produces sparse alignment on a small set of trait- bearing tokens, while activation steering pushes nearly all tokens toward the persona direction and becomes more saturated over the response.

Key Results

GCAD significantly outperforms standard residual-stream steering in long-horizon stability:

  • Coherence Stability: On a 15-trait average, standard steering saw a coherence drift of -18.6 (a massive drop), while GCAD maintained a near-flat drift of -1.9.
  • Improved Control: By the 10th turn of a conversation, GCAD raised trait expression scores from 78.0 to 93.1 compared to the baseline.
  • Trait-Specific Success: For traits like "Humorous" or "Impolite," where standard steering often leads to total coherence collapse by turn 5, GCAD allowed the model to remain both highly expressive and perfectly coherent.
Figure 4: Multi-turn coherence and trait expression on Qwen2.5-7B-Instruct for four representative persona traits. Panel A shows RLHF-opposing traits (IMPOLITE, SYCOPHANTIC); Panel B shows neutral traits (CREATIVE, HUMOROUS). In each panel, the top row reports coherence and the bottom row reports trait expression. Curves show the mean over 60 conversations with shaded ±1 standard deviation. Standard Steering is the residual-stream baseline at layer 20 (α = 2.0), while GCAD uses gated cropped attention-delta steering at layers 9–19 (cbase = 3.5, s = 1.5). GCAD maintains stable trait expression while substantially reducing the coherence collapse of standard steering.

Limitations and Future Work

  • Prompt Sensitivity: GCAD relies on the model's ability to follow the contrastive prompts used during the vector extraction phase. If a model is so heavily aligned that it refuses to "act evil" even when prompted, the extracted steering signal may be too weak.
  • Judge Bias: The authors acknowledge that using an LLM as a judge (GPT-4o-mini) might introduce biases, particularly for traits that overlap with the judge's own safety training.
  • Future Directions: The authors suggest using the GCAD gating mechanism as a diagnostic tool to monitor for "stealthy" steering or as a supervision signal for fine-tuning models to internalize these steering directions more naturally.

Summary updated

Abstract

Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Across persona-steering experiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves average coherence drift from -18.6 to -1.9 and raises turn-10 trait expression from 78.0 to 93.1. These results suggest that activation steering becomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control.
BibTeX
@misc{kang2026prompt,
  author = {Diancheng Kang and Zheyuan Liu and Ningshan Ma and Yue Huang and Zhaoxuan Tan and Meng Jiang},
  title = {{Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions}},
  year = {2026},
  eprint = {2605.10664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.10664},
}
Figure