When we fine-tune a large language model (LLM) using Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), it feels as though we are teaching the model right from wrong. When asked to generate dangerous or harmful content, an aligned model will politely but firmly refuse.
However, a growing body of research demonstrates that safety alignment is merely a superficial behavioral veneer. It acts as a late-stage conversational filter rather than a mechanism for knowledge erasure. As synthesized in The First International AI Safety Report by Yoshua Bengio, safety training typically suppresses undesirable behaviors instead of removing the underlying capabilities. The raw, pre-trained knowledge remains fully intact within the model's weights.
The Geometry of Suppression
To understand why alignment behaves as a mask rather than an eraser, we must look at how optimization algorithms alter a model's internal representations.
When a model undergoes preference optimization, it does not dismantle the neural pathways responsible for encoding harmful concepts. Instead, it learns a geometric workaround. In What Makes and Breaks Safety Fine-tuning? A Mechanistic Study, Puneet K. Dokania et al. analyzed safety fine-tuning and found that it transforms weights just enough to project unsafe inputs into a mathematical "null space." This effectively mutes the activation of harmful pathways during standard forward passes.
Similarly, Andrew Lee et al. discovered in A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity that DPO does not delete toxic features; it merely introduces a directional offset that steers the model's internal activations around "toxic neurons." The toxic machinery remains completely intact, sitting right beside the newly carved safe paths.
In Mixture-of-Experts (MoE) architectures, this phenomenon manifests as routing avoidance. As explored by Mohsen Fayyaz et al. in Steering MoE LLMs via Expert (De)Activation, alignment algorithms train the model's router to avoid activating specific "unsafe" experts, leaving the underlying harmful knowledge completely untouched inside those dormant subnetworks.
Where Latent Knowledge Persists
Probing the internal representations of aligned models reveals that the forbidden information remains highly organized and accessible across multiple dimensions of the model's internal state.
Early Processing Layers
Safety mechanisms often act as late-stage gatekeepers. In Who's asking? User personas and the mechanics of latent misalignment, Asma Ghandeharioun et al. showed that harmful capabilities remain fully latent in early and intermediate layers, even when the final layer outputs a refusal. This is because safety alignment often acts as a late-stage behavioral gatekeeper rather than a deep architectural transformation, a finding echoed by Wei Zhao et al. in Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing.
Linear Decodability
The hidden states of aligned models still encode the exact information they refuse to speak. In Linearly Decoding Refused Knowledge in Aligned Language Models, Aryan Shrivastava demonstrated that refused information can be extracted from neutral hidden states using simple linear transformations. The model still "knows" the answer; it has just been trained not to select it.
The Token Probability Tail
Even when the top-ranked token is a polite refusal (e.g., "I cannot fulfill this request"), the tokens required to generate the harmful response remain highly ranked just beneath the surface. In JULI: Jailbreak Large Language Models by Self-Introspection, Jesson Wang et al. revealed that even when a model outputs a refusal, the tokens required for a harmful response reside consistently within the top-$k$ internal probability distributions.
Peeling Back the Veneer
Because the underlying knowledge is never erased, bypassing safety guardrails is not a matter of injecting new capabilities, but rather of shifting the model's internal state back into the pre-existing harmful distributions.
This can be achieved through direct token manipulation at inference time. In Jailbreak Open-Sourced Large Language Models via Enforced Decoding, Hangfan Zhang et al. showed that by forcing the model to start its response with a compliance phrase, the decoding process bypasses the refusal layer entirely, drawing directly from the latent knowledge base. The authors demonstrated a similar vulnerability via probability manipulation in On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?.
Alternatively, internal activation steering can bypass the filter without altering the text prompt. In Uncovering Safety Risks of Large Language Models through Concept Activation Vector, Zhihao Xu et al. demonstrated that identifying and manipulating a single "safety direction" in the activation space can flip the model from refusal to compliance.
Finally, because the core knowledge structures are intact, the behavioral filter can be easily stripped away via minimal fine-tuning. Jingwei Yi et al. showed in On the Vulnerability of Safety Alignment in Open-Access LLMs that fine-tuning on a tiny amount of data can rapidly reactivate latent dangerous capabilities.
In The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models, Ann-Kathrin Dombrowski discovered that this vulnerability scales with model size: larger models exhibit a wider "safety gap," meaning they appear safer out of the box but harbor vastly more dangerous latent expertise that can be easily unlocked.
Moving Beyond Behavioral Patches
To build truly robust systems, AI safety must move beyond behavioral patches and focus on architectural or data-centric erasure.
One promising direction is Representation Noising. In Representation Noising: A Defence Mechanism Against Harmful Finetuning, David Atanasov et al. proposed destroying the underlying conceptual structures of harmful information across all model layers, ensuring that attackers cannot repurpose the model even with full weight access.
Another approach is Constrained Knowledge Unlearning, where Yucheng Zhou et al. demonstrated in Safety Alignment via Constrained Knowledge Unlearning that gradient ascent can surgically erase dangerous facts while protecting neurons essential for general intelligence.
Finally, Pratyush Maini et al. argue in Safety Pretraining: Toward the Next Generation of Safe AI that safety must be embedded during the initial pre-training phase, preventing the model from ever encoding the harmful patterns that post-hoc alignment struggles to erase.
Key Foundational Work
Jailbreak Open-Sourced Large Language Models via Enforced Decoding (2024) — contributed 3 concept elements to this direction.
Jet Expansions of Residual Computation (2024) — contributed 3 concept elements to this direction.
Who's asking? User personas and the mechanics of latent misalignment (2024) — contributed 3 concept elements to this direction.