Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks

Xin Yi, Yue Li, Linlin Wang, Xiaoling Wang, Liang He

Abstract

Ensuring safety alignment is a critical requirement for large language models (LLMs), particularly given increasing deployment in real-world applications. Despite considerable advancements, LLMs remain susceptible to jailbreak attacks, which exploit system vulnerabilities to circumvent safety measures and elicit harmful or inappropriate outputs. Furthermore, while adversarial training-based defense methods have shown promise, a prevalent issue is the unintended over-defense behavior, wherein models excessively reject benign queries, significantly undermining their practical utility. To address these limitations, we introduce LATPC, a Latent-space Adversarial Training with Post-aware Calibration framework. LATPC dynamically identifies safety-critical latent dimensions by contrasting harmful and benign inputs, enabling the adaptive construction of targeted refusal feature removal attacks. This mechanism allows adversarial training to concentrate on real-world jailbreak tactics that disguise harmful queries as benign ones. During inference, LATPC employs an efficient embedding-level calibration mechanism to minimize over-defense behaviors with negligible computational overhead. Experimental results across five types of disguise-based jailbreak attacks demonstrate that LATPC achieves a superior balance between safety and utility compared to existing defense frameworks. Further analysis demonstrates the effectiveness of leveraging safety-critical dimensions in developing robust defense methods against jailbreak attacks.

Introduction

Large Language Models (LLMs) are increasingly deployed in sensitive real-world applications, making safety alignment a top priority. However, even models aligned with techniques like Reinforcement Learning from Human Feedback (RLHF) remain vulnerable to "jailbreak" attacks. These adversarial prompts are designed to bypass safety filters by disguising harmful intent, often causing the model's internal representations to mimic those of benign queries.

Existing defense strategies often struggle with a critical trade-off: robustness versus utility. Hardening a model against attacks frequently leads to "over-defense," where the model incorrectly rejects harmless but potentially sensitive queries. Furthermore, many adversarial training methods are coarse-grained, treating all internal features as equally important, which can inadvertently degrade the model's general reasoning capabilities.

The LATPC Framework

To address these challenges, we propose LATPC (Latent-space Adversarial Training with Post-aware Calibration). This framework operates in two distinct phases to harden the model against attacks while preserving its utility for benign users.

Overview of the LATPC framework showing the two-stage defense against harmful queries and over-defense.

1. Training Phase: Latent-Space Adversarial Training

Instead of training solely on text-based inputs, LATPC intervenes directly within the model's hidden states.

  • Refusal Feature Identification: The framework uses variance analysis to identify "safety-critical" dimensions in the latent space that consistently distinguish harmful from harmless inputs. By focusing on the "Top-k" most stable dimensions, the defense becomes more surgical.
  • Simulated Attacks: During training, LATPC applies a "refusal feature removal" attack. It purposefully masks or perturbs these safety-critical dimensions to simulate an attacker attempting to suppress the model's internal safety signals.
  • Adversarial Fine-tuning: The model is fine-tuned (using LoRA) to remain steadfast and refuse queries even when its internal safety representations are being manipulated.

2. Inference Phase: Post-aware Calibration (PAC)

To mitigate the over-defense problem, LATPC introduces a lightweight calibration step during the inference process.

  • Pseudo-harmful Detection: A simple linear classifier is trained at each layer to identify "pseudo-harmful" queries—benign inputs that the hardened model might otherwise misclassify as harmful.
  • Latent Nudging: If a query is flagged as pseudo-harmful, the PAC mechanism applies a minimal mathematical adjustment to the latent representation, "nudging" it back toward the harmless feature space before the model generates a response.

Experimental Evaluation

We evaluated LATPC against eight state-of-the-art defense methods across models including Llama3-8B, Mistral-7B, and Gemma-7B. The evaluation utilized several prominent benchmarks:

  • Safety Benchmarks: HarmBench, AdvBench, and JailbreakBench were used to measure the Attack Success Rate (ASR).
  • General Capability: MMLU and TriviaQA ensured the defense did not compromise the model's intelligence.
  • Over-refusal: OR-Bench measured how often the model incorrectly rejected safe prompts.

Key Results and Insights

LATPC demonstrated a superior balance between safety and utility compared to existing frameworks.

Comparison of LATPC against state-of-the-art defense methods across safety and utility metrics.
  • Superior Robustness: On Llama3-8B, LATPC reduced the ASR of the potent "Adaptive Attack" from 91.8% to 13.8%. For "HumanJailbreaks" and "GPTFUZZER" attacks, it achieved a 0% ASR in several test cases.
  • Utility Preservation: Unlike other adversarial methods that significantly impacted general performance, LATPC maintained (and sometimes slightly improved) MMLU and TriviaQA scores.
  • Reduced Over-defense: The PAC module reduced the over-refusal rate by approximately 16% compared to standard adversarial training, making the model significantly more usable for real-world applications.

Our analysis also yielded several key insights: safety information is localized rather than uniform across dimensions; the hidden state of the final token is the most informative for detecting intent; and intervening after a decoder block is more effective than intervening at the input or within attention modules.

Conclusion

LATPC offers a robust, two-stage defense that hardens LLMs against sophisticated jailbreak attacks while effectively managing the problem of over-defense. By surgically targeting safety-critical latent dimensions and applying post-aware calibration, LATPC provides a practical path toward safer and more reliable large language models.

Summary updated

Abstract

Ensuring safety alignment is a critical requirement for large language models (LLMs), particularly given increasing deployment in real-world applications. Despite considerable advancements, LLMs remain susceptible to jailbreak attacks, which exploit system vulnerabilities to circumvent safety measures and elicit harmful or inappropriate outputs. Furthermore, while adversarial training-based defense methods have shown promise, a prevalent issue is the unintended over-defense behavior, wherein models excessively reject benign queries, significantly undermining their practical utility. To address these limitations, we introduce LATPC, a Latent-space Adversarial Training with Post-aware Calibration framework. LATPC dynamically identifies safety-critical latent dimensions by contrasting harmful and benign inputs, enabling the adaptive construction of targeted refusal feature removal attacks. This mechanism allows adversarial training to concentrate on real-world jailbreak tactics that disguise harmful queries as benign ones. During inference, LATPC employs an efficient embedding-level calibration mechanism to minimize over-defense behaviors with negligible computational overhead. Experimental results across five types of disguise-based jailbreak attacks demonstrate that LATPC achieves a superior balance between safety and utility compared to existing defense frameworks. Further analysis demonstrates the effectiveness of leveraging safety-critical dimensions in developing robust defense methods against jailbreak attacks.
BibTeX
@article{yi2026latent,
  author = {Xin Yi and Yue Li and Linlin Wang and Xiaoling Wang and Liang He},
  title = {{Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks}},
  journal = {Expert Systems with Applications},
  year = {2026},
  doi = {10.1016/j.eswa.2025.129101},
}
Figure