AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models

Aashray Reddy, Andrew Zagula, Nicholas Saban, Kevin Zhu

Abstract

AutoAdv is a novel, training-free framework that automates multi-turn jailbreaking attacks against Large Language Models (LLMs). By simulating persistent conversations rather than isolated prompts, AutoAdv achieves an Attack Success Rate (ASR) of up to 95% on models like Llama-3.1-8B, marking a 24% improvement over standard single-turn benchmarks. This paper details the architecture of AutoAdv—including its Pattern and Temperature Managers—and demonstrates how conversational persistence can erode model alignment.

Introduction

Current AI safety research predominantly focuses on "single-turn" jailbreaks, where a model's robustness is tested against a single malicious prompt. However, real-world adversaries often employ adaptive, multi-turn dialogues to gradually nudge models toward compliance. This discrepancy creates a critical "evaluation gap": safety alignment strategies optimized for one-shot refusals may fail to maintain robustness over extended interactions. AutoAdv addresses this gap by providing a scalable, automated red-teaming framework that utilizes realistic, multi-turn interactions.

Methodology

AutoAdv operates as a black-box framework, requiring no access to the target model's internal weights. It employs an "attacker LLM" (e.g., Grok 3 Mini) to drive a three-pronged strategy:

  1. Two-Phase Rewriting: The attacker disguises harmful requests within benign frames (e.g., academic exercises). If refused, it iteratively refines the prompt based on the target's feedback.
  2. Pattern Manager: This module maintains a "memory" of successful adversarial techniques, such as persona creation or domain shifting. It ranks these patterns and injects them as hints for future turns.
  3. Temperature Manager: To avoid repetitive loops, this module dynamically adjusts sampling temperature—increasing randomness during failure to explore new phrasings and stabilizing it near success.
Comparison of AutoAdv with existing attack frameworks across key features.

As shown in the comparison above, AutoAdv is uniquely positioned as a training-free, multi-turn framework that incorporates both pattern and temperature management alongside adaptive prompting.

Results and Analysis

The researchers evaluated AutoAdv using 700 harmful prompts from AdvBench and HarmBench. The results demonstrate a persistent vulnerability across various state-of-the-art models.

Multi-Turn Superiority

A key finding is the significant increase in Attack Success Rate (ASR) as the number of conversational turns ($N$) increases. For Llama-3.1-8B, the ASR jumped from approximately 71% at turn 1 to 95% by turn 6.

Attack Success Rate (ASR) across different models as the number of turns (N) increases.

Model Resilience

While all models showed vulnerability, their levels of resilience varied. Qwen3-235B reached a staggering 99% ASR within six turns. GPT-4o-mini proved the most resilient but still succumbed to an 86% ASR by the sixth turn, illustrating that even top-tier commercial models struggle with conversational persistence.

Discussion

The success of AutoAdv suggests an "erosion" of alignment: prolonged engagement allows attackers to bypass filters by maintaining a benign surface while making incremental shifts toward harmful content. The Pattern Manager's ability to treat jailbreaking as a search problem across a taxonomy of adversarial categories creates a self-improving feedback loop that becomes more effective over time.

Conclusion

AutoAdv highlights the necessity for adaptive safety mechanisms that can withstand multi-turn adversarial pressure. Future work will involve expanding the framework to multimodal and cross-lingual contexts, as well as developing defensive strategies that are as dynamic as the automated attacks they aim to prevent.

Summary updated

BibTeX
@inproceedings{reddy2025autoadv,
  author = {Aashray Reddy and Andrew Zagula and Nicholas Saban and Kevin Zhu},
  title = {{AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models}},
  booktitle = {Lock-LLM @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=JhiRuktD8v},
}
Figure