DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment

Peng Yu, Xiaoyu Wen, Zhida He, Ziyuan Zhou, Han Qi, Shao Zhang, Qiaosheng Zhang, Ying Wen, Chaochao LuarXiv 2026 · arXiv

Adversarial co-evolution—pitting an "attacker" model against a "defender" model—is the current frontier for safety-aligning Large Language Models (LLMs). However, these digital arms races often suffer from two fatal flaws: attackers become repetitive (strategy collapse), and defenders forget how to handle old tricks (adversarial forgetting).

DACE (Diversity-Driven Adversarial Co-Evolution) introduced in this paper, provides a robust framework to solve these pathologies by forcing attackers to explore a structured strategy space and equipping defenders with a "Bayesian memory" of past threats.

TL;DR

DACE improves LLM safety alignment by coupling a diversity-driven attacker reward with a Bayesian adversarial replay pool for the defender. This prevents the attacker from overfitting to a few "jailbreak" tricks and ensures the defender remains robust against both new and historical attack strategies without sacrificing general model capabilities.

What Problem Does This Paper Address?

Static safety alignment—training a model on a fixed dataset of "bad" prompts—is a losing game because the threat landscape moves too fast. Co-evolutionary frameworks, where an attacker LLM learns to break a defender LLM while the defender learns to resist, are more adaptive but prone to two "pathologies":

  1. Attack Strategy Collapse: The attacker finds a single high-reward rewrite (e.g., "Start your response with 'Sure, I can help with that...'") and overfits to it, leaving the defender blind to other types of risks.
  2. Defense Adversarial Forgetting: As the attacker moves on to new strategies, the defender’s weights shift, and it "forgets" how to defend against the earlier attacks it previously mastered.

Current remedies often focus on textual diversity (making the words different) rather than strategy diversity (making the method of attack different), and they fail to account for the uncertainty and non-stationarity inherent in a shifting adversarial game.

Figure 1: Motivation. Left: standard co-evolution collapses the attacker onto a few high-reward cells, leaving the defender trained on a biased slice of the threat landscape. Right: DACE structures the attack space into a risk-category × attack-style grid and maintains a Bayesian adversarial replay pool, so co-evolution is jointly driven by strategy-level coverage and refreshed threat memory.

Methods

DACE introduces two primary mechanisms to stabilize the co-evolutionary loop:

  • Explicit 2D Strategy Space: Instead of letting the attacker wander aimlessly, DACE defines a 120-cell grid (12 risk categories $\times$ 10 attack styles). The attacker is rewarded based on normalized marginal coverage gain. Essentially, the attacker gets a higher reward for successfully eliciting an unsafe response using a strategy that hasn't been explored much yet. This provides a non-vanishing signal that forces the attacker to "fill the grid."
  • Bayesian Adversarial Replay Pool: To stop the defender from forgetting, DACE maintains a buffer of past successful attacks. Crucially, it treats each attack as a Bernoulli trial and maintains a Beta–Bernoulli posterior for each. It uses Thompson sampling to select which old attacks to replay, balancing the need to re-verify known high-threat attacks with the need to explore uncertain ones. It also applies time decay to these counts, allowing the system to "softly forget" stale evidence as the defender's policy evolves.

The training follows a two-phase process: a strategy-conditioned warm-up via SFT, followed by iterative co-evolution using the GRPO (Group Relative Policy Optimization) algorithm.

Figure 2: Overview of DACE. Two phases: Phase 1 (Warm-up) SFTs the attacker on strategy- conditioned CoT data to produce ⟨think, strategy, answer⟩outputs. Phase 2 (Co-Evolution) alternates attacker-first: (i) attacker RL with normalized marginal coverage gain over the attack strategy space, and (ii) defender RL on a mixed batch Dtrain = Dnew ∪Dreplay of fresh and replayed attacks. The unified archive tracks both strategy frequencies (for the coverage reward) and per-entry Beta–Bernoulli threat posteriors (refreshed by per-round time decay and Thompson sampling).

Datasets

The authors utilized several standard and specialized datasets:

  • Training: Harmful seeds from Jailbreak-R1 and benign seeds from WildJailBreak.
  • Safety Benchmarks: Evaluated on HarmBench (vanilla and adversarial), WildGuardTest, DAN, OR-Bench-Toxic, XSTest, and StrongREJECT.
  • General Capabilities: To ensure no "alignment tax," they tested on IFEval, ARC-Challenge, GPQA, MMLU, and AlpacaEval 2.

Key Insights and Ideas

  • Strategy vs. Surface Novelty: The authors argue that textual novelty (Self-BLEU scores) is a poor proxy for safety. An attacker can rephrase the same "role-play" attack a thousand ways; DACE forces it to switch from "role-play" to "technical jargon" or "uncommon dialects."
  • Uncertainty Matters: By using a Bayesian approach for the replay pool, DACE distinguishes between an attack that failed once (low confidence) and one that has failed ten times (high confidence). This prevents the defender from prematurely deprioritizing dangerous attacks due to a single "lucky" refusal.
  • Non-Stationarity is a Feature: The use of time decay in the replay pool acknowledges that the "threat level" of an attack is not absolute—it changes as the defender improves.

Key Results

DACE consistently outperformed state-of-the-art baselines like MAGIC and Self-RedTeam across multiple model families (Qwen2.5, Llama3.1).

  • Superior Safety: DACE achieved near-zero Attack Success Rates (ASR) on standard benchmarks. For example, on Qwen2.5-7B, DACE reduced adversarial ASR on WildJailBreak from 0.198 (MAGIC) to 0.068.
  • OOD Robustness: DACE defenders proved significantly more robust against unseen automated attackers (like GCG, PAIR, and AutoDAN-Turbo) that were not part of the training loop.
  • No Capability Loss: DACE maintained or even slightly improved scores on general reasoning benchmarks like MMLU and AlpacaEval 2, proving that rigorous safety training doesn't have to break the model's brain.
Table 1: Safety evaluation on harmful refusal and benign compliance. For safety evaluation, lower ASR is better; higher RTA / Comply is better. Baseline results are taken from MAGIC [57].

Limitations and Future Work

  • Manual Taxonomy: The 12x10 strategy grid is manually curated based on existing safety taxonomies (Llama-Guard-4 and Rainbow Teaming). While effective, it may not capture emerging, "unknown unknown" attack vectors.
  • Judge Dependency: The framework relies heavily on the accuracy of the safety judge (e.g., Qwen3Guard). Any systematic bias in the judge could be amplified during co-evolution.
  • Expansion: The current work is limited to text-only, single-turn interactions. The authors suggest extending DACE to multi-modal, tool-using, and multi-turn agentic settings as a natural next step.

Summary updated

Abstract

Ensuring robust safety alignment of large language models (LLMs) is increasingly difficult as adversarial attacks evolve and outpace alignment pipelines built on pre-collected data. Existing co-evolutionary frameworks let the defender chase a moving attacker, yet their training dynamics expose two compounding pathologies: \emph{attack strategy collapse}, where the attacker overfits to a narrow set of high-reward rewrites, and \emph{defense adversarial forgetting}, where the defender loses competence against earlier attacks as the attack distribution drifts. Existing remedies are partial: attacker-side diversity rewards score textual novelty rather than strategy novelty, and defender-side replay relies on discrete judge scores that ignore estimation uncertainty, response stochasticity, and defender non-stationarity. We introduce \textbf{\method}, a diversity-driven adversarial co-evolution framework that targets both pathologies jointly. On the attacker side, \method couples an explicit two-dimensional strategy space (risk category $\times$ attack style) with a \emph{normalized marginal coverage gain} reward, providing a bounded, non-vanishing exploration signal at the strategy layer. On the defender side, \method maintains a unified \emph{Bayesian adversarial replay pool} whose Beta--Bernoulli threat posteriors are refreshed by time decay and sampled via Thompson sampling, anchoring defender training to an evolving threat landscape. Across standard safety, automated-attacker, and general-capability benchmarks, \method improves robustness to out-of-distribution attacks while preserving general capabilities, and yields broader attacker strategy coverage.
BibTeX
@misc{yu2026dace,
  author = {Peng Yu and Xiaoyu Wen and Zhida He and Ziyuan Zhou and Han Qi and Shao Zhang and Qiaosheng Zhang and Ying Wen and Chaochao Lu},
  title = {{DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment}},
  year = {2026},
  eprint = {2611.22278},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22278},
}
Figure