Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

Atri Vivek Sharma, Brian Formento, Alessio LomuscioarXiv 2026 · arXiv

TL;DR

StressSim is a new framework that uses genetic prompt optimization to create "difficult but realistic" simulated users for stress-testing AI agents. By evolving adversarial interaction strategies that preserve the user's original intent, StressSim exposes significant robustness gaps in frontier models—dropping success rates by up to 69%—while maintaining far higher conversation realism than previous hand-crafted methods.

What Problem Does This Paper Address?

Evaluating multi-turn AI agents (like customer service bots) is tricky. Researchers typically use other LLMs to simulate users, but these simulators suffer from "sycophancy"—they are often too cooperative, helpful, and clear, which leads to an overestimation of the agent's real-world performance.

While some researchers have tried to fix this by hand-designing "difficult" user personas (e.g., a user who is rude or gives fragmented information), these fixed strategies often fail to capture the diverse ways a real human might be difficult without being malicious. The challenge is to find a way to generate intent-preserving adversarial interactions: dialogues that are hard for an agent to handle but remain valid, natural, and focused on the original goal.

Methods

The authors introduce StressSim, a modular framework that treats adversarial simulation as a per-example constrained optimization problem. As shown in the workflow below, the system consists of three main components:

  1. Goal State Tracker: Decomposes the user's goal into sub-goals and monitors progress.
  2. Adversarial Turn Planner: Selects high-level tactics (e.g., "be rushed," "withhold info until asked").
  3. User Utterance Generator: Turns the plan into natural language.
Figure 1: Adversarial user-simulator workflow. Task instructions initialise a goal-state tracker; an adversarial planner selects turn-level user policies; and a generator produces user utterances. Planner prompts are optimised to reduce task success while preserving goal alignment.

The "secret sauce" is a genetic prompt optimization loop. Instead of fine-tuning a model, StressSim evolves the instructions given to the Adversarial Turn Planner. It generates candidate prompts, tests them in "rollouts" against the target agent, and uses an LLM-based reflection step to mutate the best-performing prompts.

The optimization objective is two-fold:

  • Maximize Agent Failure: Reduce the agent's task completion score.
  • Maintain Validity: A heavy penalty is applied if the simulated user becomes unrealistic, repetitive, or abandons the original goal.

Datasets

The researchers evaluated StressSim across three diverse benchmarks:

  • MultiWOZ: A classic multi-domain dialogue dataset for tasks like booking trains or restaurants.
  • $\tau$-bench (Retail & Airline): A more modern, complex benchmark where agents must use tools and APIs to solve user problems in retail and airline environments.

They tested three target agents: gpt-5.4-nano, kimi-k2.6, and gemini-3.8-flash.

Key Insights and Ideas

  • Intent Preservation is Key: Unlike "jailbreaking" attacks that try to make a model say something harmful, StressSim aims for "valid difficulty." The user remains a plausible customer; they just might be frazzled, cautious, or indirect.
  • Model-Specific Weaknesses: One of the most striking findings is that adversarial strategies are often model-specific. A prompt optimized to break gemini-3.8-flash might not be nearly as effective against gpt-5.6-sol. This suggests that "one-size-fits-all" robustness benchmarks are insufficient.
  • Discovery of Strategy Families: The genetic optimizer "discovered" several sophisticated tactics, such as Staged Disclosure (withholding key info until a specific dialogue stage) and Cross-turn Memory Pressure (referencing info from much earlier in the chat to see if the agent loses track).

Key Results

StressSim consistently outperformed standard simulators and hand-designed "Non-Collaborative" (NC) baselines.

  • Performance Degradation: On $\tau$-bench retail, the pass^4 score (a metric for repeated-success reliability) dropped by 69% for Gemini and 62% for GPT-5 when facing StressSim compared to a standard simulator.
  • Realism vs. Strength: As shown in Figure 2, StressSim achieves a much higher "attack strength" (relative drop in success) while maintaining superior Initial Goal Alignment (IGA)—a measure of how often the user stays on-task and realistic—compared to hand-crafted baselines like "NC tangential" or "NC emotional."
Figure 2: Attack strength versus goal align- ment, aggregated from Table 1. Each point rep- resents the mean of the evaluated metrics over the 9 dataset–LLM combinations; the x axis is the drop in passˆ4 relative to the standard user, (passˆ4standard−passˆ4sim)/passˆ4standard, and the y axis is the mean IGA.
  • Human Validation: A study with human annotators confirmed that the automated "validity judge" used during optimization was highly accurate, with a Cohen’s $\kappa$ agreement of up to 0.84 for final validity.

Limitations and Future Work

  • Reasoning Budget: The study focused on LLMs with "low reasoning budgets" to manage costs. The authors note that evaluating agents with much larger reasoning budgets (like o1-style models) remains an open area for research.
  • Subjective Rewards: The benchmarks used had deterministic, state-based rewards. In domains where "success" is more subjective, the optimization process might be vulnerable to "reward hacking" the judge.
  • Optimization Cost: While lightweight compared to fine-tuning, the process still requires roughly 20 rollouts per task instance to optimize the adversarial prompt.

Overall, StressSim provides a principled way to move beyond "happy path" evaluations and uncover the hidden failure modes of conversational AI agents.

Summary updated

Abstract

Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's performance. We here introduce StressSim, an intent-preserving adversarial user simulator framework for stress-testing agents with challenging yet valid user interactions. StressSim optimises the simulator's high-level adversarial plan for each task instance. Candidate planner prompts are evolved with a genetic prompt-optimisation loop that uses rollout traces, granular task-completion signals, and validity feedback to favour conversations that reduce task success without changing the user's underlying goal. We evaluate the approach on the MultiWOZ dataset and the retail and airline domains from tau-bench with three target LLM agents, comparing against standard simulators and hand-designed non-collaborative user behaviours. We demonstrate the effectiveness of StressSim in exposing robustness gaps across frontier LLMs, with tau-bench pass^4 scores decreasing by up to 69% on gemini-3.8-flash and 62% on gpt-5.6-sol. StressSim maintains goal preservation performance (which decreases by at most 10%), using an optimisation budget of 20 rollouts per task. Sensitivity analysis suggests that increasing the budget further decreases task success. Overall our results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism. This is achieved through lightweight prompt optimisation alone, without model fine-tuning.
BibTeX
@misc{sharma2026intent,
  author = {Atri Vivek Sharma and Brian Formento and Alessio Lomuscio},
  title = {{Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation}},
  year = {2026},
  eprint = {2610.35426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35426},
}
Figure