Latent Safety Representations in Language Models

26 concepts22 papers1 proposal

When a Large Language Model (LLM) succumbs to a jailbreak prompt or generates toxic content after adversarial fine-tuning, it is easy to assume that its safety alignment has been completely erased. However, a growing body of mechanistic analysis reveals a fascinating paradox: a model's behavioral failure to refuse a harmful prompt does not imply an internal failure to recognize it.

Instead, safety knowledge persists deep within the model's hidden states as a stable, latent representation. Understanding why these internal guardrails persist—and how they are bypassed or suppressed—has opened up a new frontier in LLM security.

The Disconnect Between Knowledge and Behavior

Why does a jailbroken model continue to comply with a harmful request even when it has been trained to refuse? The core insight of this research direction is that safety failures are typically problems of response selection rather than comprehension failures.

As demonstrated by Chujie Zheng et al. in On Prompt-Driven Safeguarding for Large Language Models, LLMs possess an inherent, latent capability to distinguish between harmful and harmless inputs within their hidden states. This internal classification occurs automatically, even in the absence of explicit safety-oriented system prompts. When an adversarial prompt bypasses a safety filter, it doesn't wipe out this awareness; it merely prevents the internal classification from governing the final token generation.

This phenomenon extends across modalities and architectures:

In short, the model "knows" the input is malicious, but this internal signal is decoupled from its behavioral output.

The Geometry of the Safety Subspace

To understand why this disconnect happens, we must look at the geometry of the model's activation space. Safety alignment typically embeds concepts of toxicity and refusal along specific, low-dimensional feature directions within the hidden layers.

In Read the Scene, Not the Script: Outcome-Aware Safety for LLMs, Zeru Shi et al. probe these hidden states and discover that middle layers often maintain highly nuanced distinctions between harmful and harmless uses of sensitive terms. However, these sophisticated internal representations are frequently overridden in the final layers by rigid behavioral policies or adversarial token distributions.

Even when a model is pushed to its limits, these internal directions remain remarkably robust. Victoria Mok Siu et al. demonstrate in COSMIC: Generalized Refusal Direction Identification in LLM Activations that even if a model is configured to refuse all inputs, the internal activations associated with refusing harmful prompts remain linearly separable from those associated with refusing harmless ones. This latent separability proves that safety-related conceptualizations are encoded independently of the final behavioral output.

[Input Prompt] ---> [Middle Layers: Latent Safety Subspace] ---> [Final Layers] ---> [Output]
                         (Accurately separates                                    (Adversarial tokens
                          harmful vs. benign)                                      override safety)

This internal monitoring happens dynamically during generation. For reasoning models that utilize an explicit chain-of-thought (CoT), Andy Arditi identified a specific "caution" direction in the activation space in Adversarial Manipulation of Reasoning Models using Internal Representations. His work shows that reasoning models evaluate safety and make refusal decisions during their internal reasoning process, illustrating a clear decoupling between diagnostic capabilities and behavioral constraints when these internal signals are manipulated.

Suppression vs. Erasure

Why do adversarial prompts or fine-tuning on toxic data cause safety failures if the underlying knowledge remains intact? The answer lies in the distinction between suppression and erasure.

Adversarial jailbreaks do not wipe out safety features; instead, they act as complex narrative distractions that overshadow them. Zhangchen Xu et al. observe in SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding that during successful jailbreaks, the internal probability for a safety refusal (such as the token "Sorry") is still present within the model's top-k token distribution but is mathematically suppressed by adversarial tokens.

A similar mechanism occurs during fine-tuning. When a model undergoes downstream fine-tuning—whether on toxic datasets or benign agentic tasks like coding—its outward refusal behavior often degrades. However, Minjun Zhu et al. demonstrate in Locking Down the Finetuned LLMs Safety that the underlying activation patterns for harmful queries are preserved. Adversarial fine-tuning merely bypasses existing safety mechanisms rather than deleting the safety-related knowledge itself.

This meta-stability is echoed by Dongyoon Hahm et al. in Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation, who show that agentic fine-tuning suppresses rather than erases safety alignment, leaving residual safety-relevant features latent within the model's internal layers. Even when a model has been explicitly fine-tuned to remove safety constraints, Jiawei Zhang et al. find in Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth that latent safety signals remain detectable in the hidden states at any depth of generation.

Reactivating and Steering Latent Safety

The persistence of these latent safety representations opens up a powerful paradigm for defense. Instead of undergoing expensive, computationally heavy retraining or alignment phases, we can design non-invasive, inference-time interventions that directly read, amplify, or steer these hidden features.

Several distinct strategies have emerged to leverage this internal awareness:

1. Decoding-Time Intervention

By monitoring hidden states during inference, defenses can dynamically adjust token distributions. Zhangchen Xu et al. amplify refusal token probabilities when latent safety signals are suppressed. Similarly, Ching-Yun Ko et al. utilize a lightweight decoding algorithm in Large Language Models can Become Strong Self-Detoxifiers to steer internal embeddings away from toxic regions of the latent space, allowing the model to act as a self-disciplined sampler.

2. Self-Activation and Prompting

Models can be prompted to look inward. Mu Li et al. introduce a framework in SAID: Empowering Large Language Models with Self-Activating Internal Defense that distills user intent and probes it with optimized prefixes to reactivate suppressed internal guardrails. In a similar vein, Canwen Xu et al. map out harmful latent regions in Leashing the Inner Demons: Self-Detoxification for Language Models by intentionally inducing toxicity to create a negative guide for steering.

3. Weight-Space Restorations

Because safety alignment is localized along specific feature directions, it can be repaired surgically. Guanhong Tao 0001 et al. show in Alleviating the Fear of Losing Alignment in LLM Fine-tuning that lost alignment can be restored by resetting a mere 0.2% of a model's weights to their pre-fine-tuned state. Taking this further, the authors of Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection show that identifying the latent refusal direction and applying a rank-one update can permanently amplify safety and re-align "uncensored" models without traditional retraining.

Ultimately, the existence of meta-stable, latent safety circuits means that safety is not a fragile veneer easily stripped away by fine-tuning or clever prompting. Instead, safety alignment fundamentally alters how language models structure information internally, leaving a permanent geometric signature that can be targeted and re-awakened to maintain robust guardrails.

Key Foundational Work

Locking Down the Finetuned LLMs Safety (2024) — contributed 2 concept elements to this direction.
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024) — contributed 2 concept elements to this direction.
On Prompt-Driven Safeguarding for Large Language Models (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_safety_awareness_192f46bc,
  title = {latent safety awareness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-safety-awareness/rdr_192f46bc357547318d8137648654cc1b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_safety_awareness_192f46bc_v1,
  title = {latent safety awareness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-safety-awareness/rdr_192f46bc357547318d8137648654cc1b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake