On Evaluating User Simulators for Interactive Agents

Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui HsieharXiv 2026 · arXiv

TL;DR

Evaluating interactive LLM agents at scale requires user simulators, but how do we know if these simulators are actually reliable? This paper introduces a "meta-evaluation" framework that uses deliberate perturbations to expose flaws in existing metrics and proposes a new three-tiered evaluation suite—measuring downstream utility, population fidelity, and trace-level intent—that is significantly more sensitive to simulator failures.

What Problem Does This Paper Address?

As LLM agents move from simple chat to complex, multi-turn tasks like coding web apps or designing workflows, evaluating them becomes a bottleneck. Recruiting real humans for every test is slow and expensive. User simulators (LLMs prompted to act like users) offer a scalable alternative, but they have a "fidelity" problem: a simulator might sound perfectly natural while being "too helpful," ignoring constraints, or approving broken code that a real user would reject.

The authors point out that we lack a rigorous way to evaluate the evaluators. If a simulator gives an agent a high score, does that reflect true agent capability or just a flaw in the simulation? Existing benchmarks often fail to detect these subtle but critical behavioral drifts, making it difficult to trust simulator-based leaderboards.

Methods

The researchers propose a two-part solution:

  1. Perturbation-Based Meta-Evaluation: To test if an evaluation metric is actually working, they deliberately "break" the simulator in 10 specific ways (e.g., making it favor a specific agent, forcing it to stop early, or changing its goal mid-stream). They then check if the metrics are sensitive enough to penalize these perturbed replays compared to a "control" version.
  2. A New Three-Tiered Evaluation Suite: They move beyond simple text similarity to a more holistic framework:
  3. L1: Downstream Ranking Agreement: Does the simulator rank a set of agents the same way real users do?
  4. L2: Population-Level Fidelity: Does the collective behavior of the simulated users (e.g., how often they ask questions vs. report bugs) match real-world distributions?
  5. L3: Trace-Level Fidelity: Does an individual simulation stay faithful to the specific user's original intent, constraints, and persona?
Figure 1: Simulator evaluation and perturbation-based meta-evaluation. Replays are compared through agent ranking agreement (L1), population-level fidelity (L2) and trace-level fidelity (L3). Paired control and perturbed replays test benchmark sensitivity and filtering.

Datasets

The study is grounded in a diverse dataset of 351 real-world multi-turn sessions involving 348 distinct users across 23 languages. These sessions involve complex artifact creation, such as building full-stack web apps, games, and automated workflows. This dataset provides the "ground truth" for how real users interact, provide feedback, and judge success.

Key Insights and Ideas

  • Metrics can be "fooled" by politeness: Existing evaluation suites (like SimulatorArena or VISTA) only detected 5–6 out of 10 deliberate perturbations. Shockingly, some composite scores actually favored simulators that were instructed to accept broken work because they remained "on-task" and polite.
  • The "Averages" Trap: Population-level metrics (L2) are necessary but insufficient. A simulator might match the overall "vibe" of a user base (e.g., average message length) while completely failing to preserve the specific constraints of an individual user (L3).
  • Agent Favoritism is Invisible to Current Metrics: All existing evaluation suites failed to detect "agent favoritism"—where a simulator gives warmer feedback to a specific model regardless of performance. This is a major red flag for using simulators to build unbiased leaderboards.
  • Fidelity vs. Utility: A simulator that is highly "realistic" (high fidelity) isn't always the most "useful" for ranking agents (high utility). The authors found that different simulator backbones excel at different levels, highlighting the need for multi-faceted evaluation.

Key Results

  • Sensitivity: The authors' proposed evaluation suite detected 10/10 perturbations, whereas prior methods from three major studies only caught about half.
  • Filtering Power: In an experiment to filter out "bad" simulations, the authors' L1 metric retained 98% of control populations while rejecting those with agent favoritism. The best existing benchmark (Sim2Real) only retained 10% under the same conditions.
  • Simulator Benchmarking: When evaluating six popular simulators (including Sonnet 5 and GPT-5-mini based versions), they found that no single simulator led across all three levels. For instance, the SimulatorArena recipe had the best agent ranking agreement (L1), but the authors' profile-based Sonnet 5 backbone showed higher population and trace-level fidelity (L2/L3).

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Reference Ranking Dependency: The L1 metric requires a "reference ranking" from real users, which is expensive to obtain for new domains.
  • LLM-as-a-Judge Bias: The framework relies on high-end LLMs (like Claude 3.5 Sonnet) to act as judges for dialogue acts and intent preservation. While meta-evaluated, these judges may still harbor their own systematic biases.
  • Static vs. Dynamic: While the perturbations are dynamic, the "real user" reference is a static snapshot. Future work could explore more interactive ways to capture the "evolving" needs of a user during a long-term project.

Not discussed in the paper: The computational cost of running the full L1-L3 evaluation suite compared to simpler metrics.

Summary updated

Abstract

User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.
BibTeX
@misc{zhou2026evaluating,
  author = {Hengguang Zhou and I-Hung Hsu and Li Chen and Priyanshi Shrivastava and Anastasios Angelopoulos and Wei-Lin Chiang and Ion Stoica and Cho-Jui Hsieh},
  title = {{On Evaluating User Simulators for Interactive Agents}},
  year = {2026},
  eprint = {2611.24892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24892},
}
Figure