Iterative Feedback Loops for Automated Red Teaming

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2408 papers511 conceptsfrozen 2026-10-04

Traditional safety testing for Large Language Models (LLMs) often relies on static "jailbreak" prompts—fixed templates designed to bypass safety filters in a single shot. However, a new wave of research suggests that the most potent threats to AI safety are not static, but adaptive. By treating red teaming as a closed-loop optimization problem, researchers are developing agents that can "learn" how to break a target model through iterative interaction and feedback.

The Shift from Static to Closed-Loop Attacks

In a static attack, an adversary tries a variety of pre-written prompts (like the "DAN" persona) and hopes one sticks. If the model refuses, the attack ends. In contrast, closed-loop red teaming treats the interaction as a multi-turn dialogue where the attacker observes the model's refusal, reflects on why it failed, and refines its strategy for the next round.

This approach is exemplified by PAIR (Prompt Automatic Iterative Refinement), introduced by Patrick Chao and Alexander Robey in Jailbreaking Black Box Large Language Models in Twenty Queries. PAIR uses an "attacker" LLM to generate a prompt, receives a response from the "target" LLM, and then uses a separate "evaluator" model to score the response. This score provides the feedback necessary for the attacker to iteratively sharpen its prompt until the target complies.

The Anatomy of an Adversarial Loop

Most modern automated red-teaming frameworks, such as FLIRT (Ninareh Mehrabi et al., FLIRT: Feedback Loop In-context Red Teaming), follow a similar structural logic:

  1. Generation: An attacker agent proposes a candidate prompt or action.
  2. Execution: The target model (or agent) processes the input.
  3. Evaluation: A judge or scanner analyzes the output for safety violations or "malicious intent fulfillment."
  4. Refinement: The attacker uses the evaluation results—and often the target's specific refusal text—to update its strategy.

This cycle allows the system to bypass defenses that might stop a one-off attempt. As Junjie Chu 0002 and Yugeng Liu demonstrate in JailbreakRadar, these dynamic, feedback-based attacks are significantly more resistant to current defense layers than fixed-template methods because they produce diverse, natural-sounding inputs tailored to the specific "blind spots" of the target.

Exploiting Agentic Reasoning and Social Dynamics

When LLMs are deployed as autonomous agents (e.g., coding assistants or financial tools), the attack surface expands. Attackers can now exploit the agent's own reasoning loops.

In the Evil Geniuses (EG) framework, Yu Tian and colleagues (Evil Geniuses: Delving into the Safety of LLM-based Agents) show that multi-agent systems are vulnerable to a "domino effect." By iteratively refining role-specific prompts, an adversarial team can compromise one agent, which then uses its trusted status to bypass the safety filters of its "teammates."

Similarly, Jinru Ding and the team behind CNFinBench (CNFinBench: A Benchmark for Safety and Compliance of Large Language Models in Finance) use an automated attacker that follows a loop of observation, reflection, and planning. This simulates social engineering: the attacker doesn't just ask for a forbidden action; it gradually erodes the model's compliance over several rounds of dialogue.

Multimodal and Tool-Based Refinement

The iterative approach is not limited to text. In the realm of GUI agents, the EVA framework (Tianjie Ju et al., EVA: Red-Teaming GUI Agents via Evolving Indirect Prompt Injection) uses behavioral feedback to evolve visual "poison." By assigning utility scores to different visual layouts and keywords based on how effectively they distract the agent, EVA discovers highly effective adversarial triggers that a human might never consider.

For agents that use tools, frameworks like SIRAJ (Ahmed Elgohary et al., SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning) analyze an agent's execution logs. If an attack fails, the system identifies whether the failure was due to a safety guardrail or a logic error, then adjusts tactics—such as injecting a sense of "urgency" or "authority"—to bypass the specific barrier in the next iteration.

The Challenge of Adaptive Defense

The primary takeaway from this research direction is that "the attacker moves second." As Milad Nasr and Nicholas Carlini argue in The Attacker Moves Second, defenses that appear robust against static benchmarks often collapse when faced with an adaptive attacker that can tailor its strategy to the specific defense mechanism.

This creates a "Reasoning-Safety Trade-off." As noted in TeleAI-Safety by Xiuyuan Chen et al., models with stronger reasoning capabilities (like DeepSeek-R1 or OpenAI o3) are actually better at jailbreaking other models—and sometimes themselves—because they can more effectively navigate the iterative planning and reflection required to circumvent complex guardrails. This "meta-jailbreaking" capability, explored in Jailbreaking to Jailbreak by Zifan Wang, suggests that the future of AI safety will be an escalating arms race between increasingly sophisticated, autonomous loops of attack and defense.

Papers in this version

Showing 60 of 408 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 218 papers
  2. Version 1 · 2026-10-04 · 408 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_prompt_refinement_iterative_ad_eceb5c7e,
  title = {iterative prompt refinement (iterative adversarial refinement)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-prompt-refinement-iterative-adversarial-refinement/rdr_eceb5c7eb294454fbad2622829335bda},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_iterative_feedback_loops_for_automated_r_eceb5c7e_v1,
  title = {Iterative Feedback Loops for Automated Red Teaming},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-feedback-loops-for-automated-red-teaming/rdr_eceb5c7eb294454fbad2622829335bda/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}