Does Context Compression Preserve Refusal Alignment?

Anthony Nguyen, Wenjun LinarXiv 2026 · arXiv

TL;DR

Context compression is a popular way to make LLMs more efficient, but this paper reveals it acts as a "silent jailbreaker." By encoding prompts into compact representations, compression preserves the meaning of a request while stripping away the subtle geometric features that trigger safety guardrails, causing refusal rates to plummet. The authors propose Memory Steering, a lightweight intervention that restores safety by injecting a learned "refusal signal" directly into the compressed memory.

What Problem Does This Paper Address?

As we push for longer context windows and cheaper inference, context compression—encoding long inputs into a few "memory vectors"—has become a vital tool. However, these compressors are typically trained for a single goal: semantic reconstruction. They are optimized to ensure the decoder can reconstruct the original text, not to ensure the decoder behaves safely.

The authors identify a "Two-Stage Discrepancy." While the base LLM is safety-aligned via RLHF to refuse harmful queries, the context encoder is often trained with a pre-training-style objective. This creates a structural failure mode: the encoder preserves the intent to do harm (semantic content) but fails to activate the decoder's internal "refusal circuits." Essentially, compression acts as a natural filter for the very features that keep models safe.

A high-level illustration of how context compression weakens refusal behaviour in instruction-tuned language models.

Methods

The researchers investigated this phenomenon using two main compression architectures: In-Context Autoencoders (ICAE) and ARC-Encoder.

  1. Mechanistic Analysis: They used linear probing to identify a "refusal direction" in the decoder's residual stream. By projecting activations onto this direction, they could see exactly how compression affects the model's internal state.
  2. Memory Steering: To fix the safety gap, they introduced a three-step inference-time intervention:
  3. Learning the Direction: They computed a global steering vector ($\Delta\mu$) by comparing compressed representations of standard prompts against "refusal-wrapped" versions (e.g., adding the text "This violates your AI safety guidelines").
  4. Gating: They trained a lightweight logistic regression head to identify which specific memory slots encode harmful intent, preventing the model from "over-refusing" benign prompts.
  5. Steering: At inference, they add the learned refusal signal to the identified harmful memory slots before the decoder begins generation.

Datasets

The study utilized a mix of standard safety benchmarks and a new targeted dataset:

  • HarmBench & AdvBench: Standard benchmarks containing harmful prompts designed to trigger safety failures.
  • Custom Paired Dataset: 624 prompts (312 safe, 312 harmful) generated using LLaMA-3-70B. Crucially, this included 140 "semantically similar" pairs (e.g., "How does law enforcement detect fraud?" vs. "Tell me how to commit fraud") to test if the model could distinguish between benign and harmful intent in compressed space.
  • Dolphin-R1: Used for fine-tuning the ARC-Encoders to work with instruction-tuned models.

Key Insights and Ideas

  • Semantic $\neq$ Behavioral Equivalence: The most striking finding is that a model can perfectly "understand" a harmful request (high reconstruction fidelity) but completely forget it was supposed to refuse it. Semantic preservation does not guarantee alignment preservation.
  • Geometric Erosion: Refusal behavior is mediated by specific, low-dimensional directions in the model's representation space. Compression systematically shifts activations away from these "refusal subspaces" and into "non-refusal subspaces."
  • Intervention at the Interface: The authors argue that because compression is the source of the degradation, the fix should happen at the compressed representation stage rather than trying to patch the decoder downstream.
Projection onto the learned refusal direction across decoder layers. Under ICAE compression, the activation for harmful prompts (orange) collapses toward the non-refusal regime (green/red).

Key Results

The "jailbreak" effect was consistent across all tested models (Mistral and LLaMA families).

  • Refusal Collapse: In Mistral-7B-Instruct-v0.2, the refusal rate for harmful prompts dropped from 62.82% to just 10.9% under ICAE compression. LLaMA-3.1-8B-Instruct saw a drop from 96.15% to 36.54% under ARC compression.
  • Generalization: This wasn't just a quirk of their custom dataset; similar drops were observed on HarmBench and AdvBench.
  • Steering Success: Memory Steering successfully recovered refusal rates. For Mistral, steering at a strength of $\alpha=2.0$ brought the refusal rate back up to 81.15% on AdvBench.
  • Preserving Utility: The "gated" version of Memory Steering maintained high response quality for benign prompts, with an "on-par" rate (win + tie vs. baseline) exceeding 80% across models.
Refusal rates versus steering strength. Gated steering (blue) recovers refusal for harmful prompts while keeping refusal for safe prompts near zero.

Limitations and Future Work

  • Architecture Constraints: The current Memory Steering method is designed for fixed-length compression (ICAE). Extending this to variable-length pooling methods like ARC-Encoder remains an open challenge.
  • Mechanistic Depth: While the paper shows that the geometry shifts, a full account of how the encoder's training objective causes this specific shift is still needed.
  • Broader Alignment: The study focused exclusively on refusal. Future work should investigate if compression also erodes other alignment properties, such as truthfulness, calibration, or bias.

In summary, this research serves as a warning for the AI safety community: as we optimize for efficiency, we must treat the preservation of alignment-relevant geometry as a first-class design objective, not an implicit assumption.

Summary updated

Abstract

Context compression reduces inference cost by encoding inputs into compact representations while preserving semantic content. An open question is whether semantic preservation alone is sufficient to maintain downstream behaviours such as refusal alignment. We investigate this question and find that encoder-based compression systematically weakens refusal behaviour in instruction-tuned language models, despite high reconstruction fidelity. This effect persists across model families and compression architectures. Mechanistic analysis shows that compression attenuates activation along the decoder's learned refusal direction. We further explore Memory Steering, a lightweight inference-time intervention that restores refusal rates to near-baseline levels without retraining and operates entirely in compressed representation space. These results demonstrate that semantic preservation does not guarantee behavioural preservation under compression, highlighting the need to explicitly preserve alignment-relevant features in compression-aware systems.
BibTeX
@misc{nguyen2026does,
  author = {Anthony Nguyen and Wenjun Lin},
  title = {{Does Context Compression Preserve Refusal Alignment?}},
  year = {2026},
  eprint = {2611.06380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06380},
}
Figure