Safety Alignment as a Superficial Output Filter

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2169 papers243 conceptsfrozen 2026-10-04

When we "align" a Large Language Model (LLM) to be safe, we typically use techniques like Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) to teach the model to refuse harmful requests. However, a growing body of research suggests that these methods do not actually remove harmful knowledge from the model. Instead, they act as a superficial behavioral veneer—a thin layer of refusal logic that masks, but does not eliminate, the dangerous capabilities latent within the model's weights.

The Suppression vs. Erasure Problem

The fundamental issue is the distinction between suppression and erasure. During pretraining, models ingest vast amounts of data, including potentially toxic or dangerous information. Standard safety alignment focuses on the model's output behavior: it rewards the model for saying "I cannot fulfill this request" and penalizes it for providing harmful content.

As ZHUO ZHANG and colleagues argue in Make Them Spill the Beans!, this process merely reduces the probability of harmful tokens appearing as the top-ranked choice. The underlying knowledge remains intact in the model's parameters. Weikai Lu notes in the Eraser study that traditional alignment creates a "refusal layer" while leaving the technical expertise (e.g., how to synthesize a pathogen) fully accessible in the latent space.

Why the Veneer is Brittle

Because the safety layer is a behavioral wrapper rather than a structural change, it can be bypassed through several "coercive" or adversarial means:

1. Minimal Fine-Tuning (Shadow Alignment)

If safety is just a thin layer, it should be easy to strip away. Xianjun Yang demonstrated this with Shadow Alignment, showing that as few as 100 malicious examples and one hour of GPU time can "unalign" a model like Llama-2. Similarly, Rishabh Bhardwaj introduced Unalignment, a "parametric red-teaming" method that reveals how minimal parameter updates can expose the biases and harms hidden beneath the surface.

2. Exploiting the Probability Distribution (Logits)

Even when a model's top-ranked token is a refusal (e.g., "I"), the rest of the probability distribution often still contains the "forbidden" knowledge. Jesson Wang showed in JULI that over 85% of the tokens needed for a prohibited response remain in the top-5 most likely predictions during a refusal. By looking at these "shadow" logits, researchers like Yuan Zhou have been able to extract private data that the model was trained to hide.

3. Latent Activation Steering

Safety filters often operate at the final stages of computation. Asma Ghandeharioun found that harmful content is often fully formed in the model's intermediate layers before being suppressed in the final output. By applying Activation Steering Attacks (ASA), Zongqi Wang demonstrated that tiny perturbations to internal hidden activations can bypass safety filters entirely, proving that the "safety" does not extend to the model's internal representational space.

The Intelligence Paradox

A particularly striking finding by Youliang Yuan in CipherChat is that more intelligent models are often less safe. Because GPT-4 is highly capable of reasoning in ciphers or ASCII, it can be induced to bypass its own natural-language safety rules. The model understands the concept of the harm but fails to generalize its refusal behavior to non-standard encodings, a phenomenon known as "mismatched generalization."

Moving Toward Robust Safety

The consensus across these 136 papers is that behavioral alignment is fundamentally limited. Noam Wies provides a mathematical framework in Fundamental Limitations of Alignment, proving that if a behavior exists in the pretraining distribution, a sufficiently long adversarial prompt can almost always trigger it.

To build truly safe models, researchers are exploring more permanent interventions:

  • Machine Unlearning: Weikai Lu's Eraser and Yucheng Zhou's Constrained Knowledge Unlearning attempt to surgically remove harmful neurons or gradients.
  • Pretraining Data Filtering: Kyle O'Brien demonstrates in Deep Ignorance that removing dual-use data (like bioweapon instructions) during pretraining creates "tamper-resistant" models that cannot be easily re-aligned for harm.
  • Latent Adversarial Training: Abhay Sheshadri suggests Targeted LAT, which trains models to resist attacks on their internal "thought processes" rather than just their input text.

By shifting the focus from what the model says to what the model knows, the field aims to move beyond the "cat-and-mouse" game of prompt engineering toward models that are inherently incapable of generating catastrophic harm.

Papers in this version

Showing 60 of 169 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 86 papers
  2. Version 1 · 2026-10-04 · 169 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_superficial_safety_alignment_b57a074f,
  title = {superficial safety alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/superficial-safety-alignment/rdr_b57a074f99b547f9a9d5037f80abc713},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_safety_alignment_as_a_superficial_output_b57a074f_v1,
  title = {Safety Alignment as a Superficial Output Filter},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/safety-alignment-as-a-superficial-output-filter/rdr_b57a074f99b547f9a9d5037f80abc713/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}