The Alignment Robustness Gap in Large Language Models

54 concepts48 papers1 proposal

When we train large language models (LLMs) to be safe, we typically use techniques like Reinforcement Learning from Human Feedback (RLHF) or Direct Policy Optimization (DPO). These methods successfully guide models to be helpful and harmless under standard conversational conditions. However, a profound vulnerability exists at the core of modern AI safety: the alignment-robustness gap. As demonstrated in Are aligned neural networks adversarially aligned? by Anas Awadalla et al., ensuring a model behaves well for honest users is fundamentally different from securing it against adversarial manipulation.

Suppression vs. Elimination

To understand why this gap exists, we must look at what safety alignment actually alters within a model's weights. Standard alignment does not erase a model's underlying capabilities or the harmful information it absorbed during pre-training. Instead, as argued in Vulnerabilities of Large Language Models to Adversarial Attacks by Yu Fu et al., it merely creates a superficial layer of protection that suppresses harmful outputs rather than removing the underlying capacity to generate them.

This superficiality leads to a phenomenon known as "fake alignment." In Fake Alignment: Are LLMs Really Aligned Well?, Yixu Wang et al. show that while models learn to adopt a "safe" response style in open-ended text, they often fail to make safe decisions when evaluated via alternative formats like multiple-choice questions. The underlying hazardous knowledge remains intact; it has simply been papered over by a stylistic filter.

Competing Objectives and Mismatched Generalization

Why is this superficial filter so easy to bypass? In Jailbroken: How Does LLM Safety Training Fail?, Alexander Wei et al. break down safety failures into two primary structural causes: competing objectives and mismatched generalization.

LLMs are trained to fulfill two conflicting goals simultaneously: to be maximally helpful (fulfill the user's prompt) and to be safe (refuse harmful requests). Adversarial prompts—or "jailbreaks"—exploit this tension by framing a harmful request within a complex, high-priority context, such as a hypothetical roleplay, an intricate coding challenge, or a multi-turn logical trap. Because the model generalizes its helpfulness objective across these complex structures better than its safety constraints, the helpfulness objective overrides the safety filter. The model is tricked into prioritizing its duty to assist over its duty to protect.

The Vulnerability of Modality and Scale

This fragility becomes even more pronounced when models interact with new modalities or operate at smaller scales. In multimodal large language models (VLMs), the alignment-robustness gap can be exploited effortlessly. Xiangyu Qi et al. demonstrated in Visual Adversarial Examples Jailbreak Aligned Large Language Models that adding mathematically optimized, imperceptible perturbations to an image can reliably jailbreak text-aligned models. The visual processing layers act as a backdoor, bypassing the text-based safety filters entirely because the safety alignment was not robustly integrated across modalities.

Conversely, scaling down models introduces a different illusion of safety. In Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Models, Sibo Yi et al. discovered that small language models (SLMs) often appear robust against complex jailbreaks simply because they lack the reasoning capacity to comprehend the adversarial intent—a state of "safety through incapacity" rather than true alignment.

Moving Toward Inference-Time Defenses

Because training-time alignment relies on a static distribution of prompts, it leaves predictable boundaries that attackers can map out and exploit. To counter this, researchers are shifting toward dynamic, inference-time interventions.

For instance, in Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment, Soumya Suvra Ghosal et al. propose shifting safety alignment to inference-time using controlled decoding guided by a safety reward model. Similarly, in Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models, Ran He introduces a test-time immunization framework that detects novel jailbreak patterns during inference and dynamically adapts the model on the fly. By moving from static training filters to adaptive inference defenses, the field aims to bridge the gap between superficial alignment and true adversarial robustness.

Key Foundational Work

Are aligned neural networks adversarially aligned? (2023) — contributed 2 concept elements to this direction.
Finding and Fixing Undesirable Behaviors in Pretrained Language Models (2022) — contributed 1 concept elements to this direction.
Jailbroken: How Does LLM Safety Training Fail? (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_alignment_robustness_gap_97a6de02,
  title = {alignment-robustness gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/alignment-robustness-gap/rdr_97a6de02b48c44459f98db23e05858e8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_alignment_robustness_gap_97a6de02_v1,
  title = {alignment-robustness gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/alignment-robustness-gap/rdr_97a6de02b48c44459f98db23e05858e8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake