Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks

Xin Yi, Yue Li, Dongsheng Shi, Linlin Wang, Xiaoling Wang, Liang He

TL;DR

This paper introduces LATPC, a novel defense framework for Large Language Models (LLMs) that combines adversarial training in the latent space with a post-inference calibration step. LATPC effectively reduces the success rate of jailbreak attacks while simultaneously mitigating the problem of "over-defense," where LLMs incorrectly reject benign queries.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly powerful, but their widespread adoption is hindered by safety concerns. One major challenge is "jailbreaking," where attackers craft malicious prompts to bypass the LLM's safety guardrails and elicit harmful or inappropriate content. While adversarial training has shown promise in defending against these attacks, a significant drawback is "over-defense." This occurs when the model becomes too cautious and starts rejecting legitimate, benign queries, severely impacting its usefulness. Existing methods often struggle to strike a balance between robust defense and maintaining model utility.

Methods

LATPC tackles this problem in two main phases:

  1. Latent-Space Adversarial Training (Phase 1):
    • Refusal Feature Identification: The core idea here is to identify the "safety-critical" dimensions within the LLM's internal representations (latent space). The authors achieve this by analyzing the differences in latent activations between harmful and harmless queries. They use a variance-based method to pinpoint dimensions that consistently contribute to the distinction between safe and unsafe responses.
  2. Mask-Based Refusal Feature Removal Attack: This simulates how jailbreak attacks work by targeting and removing these identified refusal features from the latent space of harmful queries. This forces the model to learn to resist these specific types of manipulations.
  3. Adversarial Training: The LLM is then trained on this modified data, learning to generate safe responses even when its refusal features are under attack. This phase focuses on enhancing the model's inherent robustness.
  4. Post-Aware Calibration (Phase 2):
    • Addressing Over-Defense: This phase is applied during inference (when the LLM is actually responding to user queries). It aims to correct the over-defense issue introduced by adversarial training.
  5. Embedding-Level Calibration: The authors train a linear classifier to distinguish between pseudo-harmful (mistakenly rejected benign) queries and truly harmless queries. Based on this classification, they adjust the latent representations of pseudo-harmful queries to bring them closer to the benign query space. This subtle adjustment helps the model recover its ability to respond to legitimate queries without compromising its safety against actual attacks.

Datasets

The paper uses several datasets for different purposes:

  • Training Refusal Features: Arditi et al. (2024) dataset, containing harmful and harmless queries.
  • Adversarial Training: A combination of 5,000 harmful and 5,000 harmless requests from Zou et al. (2024), supplemented with samples from OR-Bench (Cui et al., 2024). Harmless conversational queries are from UltraChat (Ding et al., 2023).
  • Post-Aware Calibrator Training: Harmless queries from the validation set of Arditi et al. (2024) and pseudo-harmful queries curated from OR-Bench.
  • Evaluation: Three standard benchmarks are used: HarmBench (Mazeika et al., 2024), AdvBench (Zou et al., 2023b), and JailbreakBench (Chao et al., 2024). The paper also uses specific attack types like GCG, PAIR, HumanJailbreaks, GPTFUZZER, and Adaptive Attack for evaluation.

Key Insights and Ideas

  • Latent Space is Key: The paper strongly emphasizes that understanding and manipulating the LLM's internal latent representations is crucial for both crafting effective attacks and developing robust defenses.
  • Targeted Defense: Instead of a blanket approach, LATPC focuses on identifying and leveraging "safety-critical" latent dimensions. This targeted approach is more efficient and less likely to disrupt the model's general capabilities.
  • Over-Defense is a Trade-off: The authors highlight that adversarial training, while effective against attacks, often leads to over-defense. Their post-aware calibration is a clever way to mitigate this specific side effect without reintroducing vulnerabilities.
  • Variance-Based Feature Selection: The variance-based method for identifying safety-critical dimensions is shown to be more robust and balanced than simple magnitude-based selection, leading to better defense performance.

Key Results

  • Reduced Attack Success Rate (ASR): LATPC significantly reduces ASR across various jailbreak attacks. For instance, on HarmBench, ASRs for HumanJailbreaks and GPTFUZZER are reduced to 0%.
  • Minimized Over-Refusal Rate (ORR): The post-aware calibration effectively reduces the ORR, with PAC decreasing ORR by 16.1% in Llama3-8B while maintaining comparable general capability.
  • Balanced Performance: LATPC achieves a superior balance between safety and utility compared to existing defense frameworks, outperforming them across ASR, general capability, and over-refusal rate.
  • Model Agnostic: The framework shows effectiveness across different LLM architectures (Llama3-8B, Mistral-7B, Gemma-7B, Qwen2-7B).

Limitations and Future Work

  • White-Box Access Required: A primary limitation is that LATPC requires white-box access to the LLM's internal representations and the ability to fine-tune its parameters. This makes it unsuitable for strict black-box scenarios where only API access is available.
  • Ethical Considerations: The research involves generating and analyzing potentially harmful prompts, which requires careful ethical handling and controlled environments.
  • Future Work: Extending LATPC to pure black-box defenses is suggested as a promising direction for future research. The authors also emphasize the need for continuous monitoring and updates for LLM safety.

Summary updated

Abstract

Ensuring safety alignment is a critical requirement for large language models (LLMs), particularly given increasing deployment in real-world applications. Despite considerable advancements, LLMs remain susceptible to jailbreak attacks, which exploit system vulnerabilities to circumvent safety measures and elicit harmful or inappropriate outputs. Furthermore, while adversarial training-based defense methods have shown promise, a prevalent issue is the unintended over-defense behavior, wherein models excessively reject benign queries, significantly undermining their practical utility. To address these limitations, we introduce LATPC, a Latent-space Adversarial Training with Post-aware Calibration framework. LATPC dynamically identifies safety-critical latent dimensions by contrasting harmful and benign inputs, enabling the adaptive construction of targeted refusal feature removal attacks. This mechanism allows adversarial training to concentrate on real-world jailbreak tactics that disguise harmful queries as benign ones. During inference, LATPC employs an efficient embedding-level calibration mechanism to minimize over-defense behaviors with negligible computational overhead. Experimental results across five types of disguise-based jailbreak attacks demonstrate that LATPC achieves a superior balance between safety and utility compared to existing defense frameworks. Further analysis demonstrates the effectiveness of leveraging safety-critical dimensions in developing robust defense methods against jailbreak attacks.
BibTeX
@article{yi2026latent,
  author = {Xin Yi and Yue Li and Dongsheng Shi and Linlin Wang and Xiaoling Wang and Liang He},
  title = {{Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks}},
  journal = {Expert Systems with Applications},
  year = {2026},
  doi = {10.1016/j.eswa.2025.129101},
}
Figure