Customer-service and tech-support agents are now routinely scored by pairing them with a language model that plays the customer. Such a model is called a user simulator. The trouble is that the simulated customer is usually a strong model. It reads every instruction perfectly, never fumbles a settings menu, and never forgets its own order number. So the benchmark mostly measures how hard the task is for the agent. It largely ignores how hard the task is for the person on the other end. Our premise is that the gap between simulated and real success is not spread evenly across tasks. It piles up on tasks that are hard for the user: long instruction sequences, actions the user must take themselves, and information the user must find and relay. If that user-side difficulty can be predicted cheaply from the simulator, then a small set of human sessions is enough to correct a synthetic benchmark task by task. The result is an estimate of how agents will fare with real people, with honest error bars, instead of a single inflated pass rate.
The proposal borrows its main tool from educational testing. There, "item difficulty" is the probability that a typical examinee fails a question, and it is estimated with item response theory, a statistical model that places people and questions on a shared ability–difficulty scale. That literature has learned the hard way that strong language models cannot simply be asked how hard something is for humans. Instead, it builds populations of deliberately weaker or perturbed model examinees and watches where they fail. We carry this idea from exam questions into multi-turn agent tasks. We then treat the resulting user-side difficulty score as the variable that governs the synthetic-to-real correction.
Core Claim
For a task $t$, let $s_t^{\text{sim}}$ be an agent's success rate when paired with a standard simulated user, and $s_t^{\text{hum}}$ its success rate when paired with real people. We claim the per-task gap $g_t = s_t^{\text{sim}} - s_t^{\text{hum}}$ varies a great deal across tasks. We further claim that most of this variation is explained by a user-side difficulty score $b_t$. This score is estimated without any human data, from how much a task's success drops as the simulated user is systematically weakened. The calibrated estimate of human success is
$$\hat{s}_t^{\text{hum}} = s_t^{\text{sim}} - f(b_t, x_t),$$
where $x_t$ are cheap task descriptors (number of user-side actions, instruction length), and $f$ is a small regression fitted on a human "seed" subset of tasks. The claim is falsifiable in a sharp way. If a constant correction $f = c$ performs as well as the difficulty-dependent one, the idea is wrong and the gap is a uniform tax rather than a task-specific one.
Where This Sits
Three lines of work motivate the proposal. The first is the curse-of-knowledge result. Li et al. benchmarked over 20 LLMs as judges of human difficulty on USMLE, SAT and Cambridge items and found an average Spearman correlation of 0.28 with real student difficulty, falling to about 0.13 on USMLE, where 70.4% of the items hardest for humans were solved by more than 90% of models. That study covers static exam items. We hypothesise that the same blindness makes simulated users unrealistically competent in dialogue.
The second is τ²-bench. Barres et al. gave the simulated user its own tools in a telecom troubleshooting domain. They measured a roughly 20% "coordination tax" relative to a No-User mode, in which the agent controls everything itself. GPT-4o dropped from 74% in retail to 34% in telecom. That tax is measured with a simulated user. Nobody knows how large it is with a confused human, or which tasks it falls on.
The third is the general record of synthetic successes collapsing on real data. Trirat et al. report a classifier at 99.7% on synthetic MOOC logs that failed on real logs. Sajjadi et al. show that models robust to simulated Gaussian grading noise failed on real misconception-driven errors. Cheng et al. show agents gaming a simulator to reach 97.1% success with unnatural language.
Several papers supply the method. LLaSA built a "student body" of 65 LLMs of varying size and fitted a Rasch item response model to estimate question difficulty. Duenas et al. degraded a model examinee through stem mutilation and temperature changes, fed the resulting failure features into ridge regression, and reached second place in the BEA 2024 USMLE task (RMSE 0.303). Epi2Diff turned reasoning-model traces into sequences of cognitive episodes such as verifying and backtracking, and used them to predict human item difficulty. We reuse all three ideas, but the examinee becomes the simulated user inside a closed-loop agent episode rather than a model answering a single multiple-choice question.
Zotos constrains the design. Large models become insensitive to answer-order perturbation, so uncertainty signals from a single strong model lose resolution. We therefore weaken the user through model capacity and behavioural policies, not only through sampling noise.
ECHO supplies the evidentiary frame. ECHO argues that synthetic users count as evidence only when they are bounded by governed human seed data, carry uncertainty, and raise rejection flags. It offers no implementation. Ours is a concrete, testable instance of that protocol for agent benchmarks.
Persona Policies (Chopra et al.) are the closest existing work. They evolve Python generators that make τ²-bench simulated users impatient, vague or reluctant while keeping task goals fixed. They reach 80.4% human-likeness in blinded ratings and train agents that are 17% more robust in relative task success. PPol makes the simulator look and act more human across the board. It does not ask which tasks the simulator gets wrong, and it does not output a calibrated estimate of human success.
Research Program
Synthetic user population. The first component is a graded population of simulated users for τ²-bench telecom (114 tasks, dual control) and τ-retail (115 tasks; Yao et al., code). The population crosses three factors. The first is user-model capacity, spanning small open models to frontier models, following LLaSA. The second is behavioural policy: plain prompts and PPol-style personas. The third is instruction degradation in the spirit of Duenas et al.: the simulated user sees a lossy paraphrase of the agent's instruction, or must recall a detail from many turns earlier.
Separating agent-side from user-side difficulty. Each population member is paired with a fixed set of three agents, and every task is also run in No-User mode. From the No-User runs we estimate agent-side difficulty. From the with-user runs we fit a two-facet item response model with user ability, agent ability, and a task's user-side difficulty $b_t$. The No-User runs pin down the agent facet, which makes the user facet identifiable. As a secondary signal, we extract Epi2Diff-style episode features from reasoning-capable simulators, such as rereading or self-correction before acting on an instruction. This tests whether process features add predictive value beyond outcome-based difficulty.
Human data. The second component is a human study. Paid participants play the user role on telecom and retail tasks, paired with the same three agents through the τ²-bench interface. Their user-side tools are exposed as a simple mock phone UI. We record database-verified task success, user-side action errors, turns, time, abandonment, and a short post-task effort rating. The study targets roughly eight human sessions per task–agent cell on a subset of agents and enough coverage to fit per-task human success.
Calibration protocol. The third component follows ECHO. Tasks are split into a human seed set and a held-out set. The correction $f$ is fitted on seed tasks only. Held-out predictions carry bootstrap intervals. A task is flagged for rejection when its $b_t$ or $x_t$ falls outside the seed range, or when the population fit is poor.
Baselines
The naive reference point is raw simulated success $s_t^{\text{sim}}$ from the default τ²-bench simulator, which is what leaderboards currently report. On τ-airline the repository leaderboard shows claude-3-5-sonnet-20241022 at pass^1 0.460 (resource). We found no reported result on Lacuna for human-paired success on any τ-bench domain, so this proposal will produce the first such reference.
The strongest competitor for making simulated success match human success is PPol (Chopra et al.). Its reported numbers are 80.4% human-likeness and a +17% relative robustness gain. We run PPol-conditioned simulators with the same agents, tasks and sampling budget as our population. If the released generators are unavailable, we reimplement the persona prompts from the paper and say so. Our method must achieve lower held-out per-task error against human success than PPol's raw simulated success. We also report PPol plus our correction, because the two are complementary rather than exclusive.
The strongest calibration competitor is uniform correction. It uses the same human seed tasks to subtract a constant (or agent-specific constant) gap, in the style of standard prediction-powered estimation. It receives exactly the same human data, and it is the ablation that isolates our key ingredient. Our method beats it only if the difficulty-dependent $f$ reduces held-out mean absolute error in per-task human success, and improves interval coverage, by a margin that a paired bootstrap over tasks shows is not noise.
For the difficulty signal itself, we compare against two alternatives. One is an LLM judge asked to rate each task's difficulty "for an ordinary customer"; Li et al. report a Spearman correlation of 0.28 for this observer setup on exam items. The other is a Lasso over cheap task text features, following ITEC, whose Lasso ranked second of 34 on BEA response-time prediction. Both are scored by Spearman correlation with human user-side difficulty and as alternative inputs to $f$.
A second ablation replaces the graded population with many samples from a single strong simulator. This tests whether weakening the user, rather than mere repetition, is what produces a useful $b_t$.
Why not just…?
Why not just make the simulator realistic with PPol and trust its numbers? PPol optimizes how human the user seems and how many behaviour types it covers. A persona can be impatient and terse yet still carry out a six-step SIM-reseating instruction flawlessly, because the underlying model has no trouble with the instruction. This is exactly the curse-of-knowledge pattern Li et al. document. The PPol baseline directly tests whether PPol alone removes the per-task gap. If it does, that is a genuine and publishable negative result for our method. We would then reframe the contribution as validating PPol's realism at the task level.
Why not just run a small human study and subtract the average gap? This is the real threat. If $g_t$ is roughly constant across tasks, uniform correction is simpler and equally good. Our core claim is the opposite, and the uniform-correction ablation decides it. Barres et al.'s observation that success approaches zero once required actions exceed 7 suggests the gap is task-dependent, but that observation comes from simulated users only.
Why not just ask a strong LLM which tasks are hard for users, or count instruction steps? Direct difficulty rating reached only ρ = 0.28 on exam items (Li et al.). Behavioural simulation beat direct prompting in the BEA task (Duenas et al.). Simple step counts could still capture much of the signal in multi-turn tasks. That is why the Lasso over task features is a full baseline and part of $x_t$, and why we report how much $b_t$ adds beyond it.
Why not just use agent-side difficulty from pass^k across many agents? Agent failure rates mix agent difficulty with user difficulty. The No-User runs separate the two. If agent-side difficulty alone predicts $g_t$ as well as $b_t$ does, the user simulation adds nothing, and the experiment will show that.
Decisive Checks
The first check is whether $g_t$ varies across tasks beyond sampling noise. We test this with a variance-components fit on the human data. Without real heterogeneity the proposal fails at the outset.
The second check is whether synthetic $b_t$ ranks tasks by human user-side difficulty, measured as the drop from No-User success to human-paired success. It must do so better than the LLM judge and the text-feature Lasso.
The third check is whether the difficulty-dependent correction beats uniform correction on held-out tasks and on a held-out agent. We test this in a leave-one-agent-out split, so the correction is shown to transfer to an agent never paired with humans. That transfer is the practical use case.
The fourth check is whether calibrated estimates reorder agents relative to raw leaderboards. Agents that coordinate well with confused users should rise, consistent with the coordination tax in Barres et al. and the incremental-information penalties in Laban et al..
Finally, we check that ECHO-style rejection flags actually mark the tasks with the largest held-out errors. Without that, the flags would be decoration rather than evidence.
Workshop authors to connect with
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents: Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction: Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou
- ECHO: Evidence-Calibrated Human Outcomes from Synthetic Users for Interaction Design: Dylan Xinming Hou
References
- Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- IntelliMOOC: Intelligent Online Learning Framework for MOOC Platforms
- Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms do not Improve over Simple Baselines
- Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator
- Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation
- UPN-ICC at BEA 2024 Shared Task: Leveraging LLMs for Multiple-Choice Questions Difficulty Prediction
- From Objectives to Questions: A Planning-based Framework for Educational Mathematical Question Generation
- Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?
- ECHO: Evidence-Calibrated Human Outcomes from Synthetic Users for Interaction Design
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
- {$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains
- ITEC at BEA 2024 Shared Task: Predicting Difficulty and Response Time of Medical Exam Questions with Statistical, Machine Learning, and Language Models
- LLMs Get Lost In Multi-Turn Conversation
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction