Conversational agents for customer service and similar work are mostly tested against language models pretending to be users. Those stand-in users are too cooperative. On the τ-bench customer-service tasks, simulated users let agents succeed up to 77.8% of the time, compared with 63.6% when real people play the same roles (Mind the Sim2Real Gap in User Simulation for Agentic Tasks). The field's usual response is to build better simulators. A second question gets less attention: is the gap a uniform discount, which one correction factor could fix, or is it concentrated in particular tasks, where a polite simulator quietly fills in information a real user would withhold? If it is concentrated, can we tell from the task's written description which tasks those are, before spending money on human trials?
This note proposes to treat each benchmark task as a test question in the sense of item response theory (IRT). IRT is the statistical model behind standardized testing. It explains each success or failure through two kinds of quantities: a skill level for the test-taker (here, the agent) and properties of each question, chiefly its difficulty and its discrimination, meaning how sharply it separates stronger agents from weaker ones. Using a crossed human-and-simulator study, we would measure how much each task's difficulty and discrimination change when a simulator replaces a real user. We would then train a model that reads the task's written specification and predicts that per-task shift. If this works, the simulator gap stops being an unexplained bias in leaderboard numbers. It becomes a property of each task, and it can be estimated and corrected for new tasks with few or no human rollouts.
Core Claim
The sim-to-real gap in user-simulated agent benchmarks has two components. The first is task-specific difficulty drift: the simulator makes some tasks much easier than humans do, and others not at all. The second is discrimination compression: tasks separate agents of different skill less sharply under a simulator than under humans. We claim the first component is large, predictable from task text, and correctable. We expect the second to be partly irreducible by offset corrections, so it must be measured rather than adjusted away.
Concretely, for agent $j$, task $i$ and user source $s$ (a human, or a particular simulator), we model the probability of success as
$$P(\text{success}_{jis}) = \sigma\big(a_i^{(s)}(\theta_j - b_i - \delta_i^{(s)} - u_{s})\big),$$
where $\sigma$ is the logistic function, $\theta_j$ is the agent's skill, $b_i$ is the task's difficulty with real humans, $\delta_i^{(s)}$ is the task-specific difficulty shift caused by source $s$ (fixed to zero for humans), $u_s$ is a per-source leniency term (or, for individual human participants, their personal severity), and $a_i^{(s)}$ is the task's discrimination under that source.
This borrows the idea of a separate "rater" facet from the many-facet Rasch model, which Correcting Human Labels for Rater Effects in AI Evaluation used to separate summary quality from the severity of human raters. Here the facet is the user, and it varies by task. The new prediction target is $\delta_i^{(s)}$, estimated from the text of the task instruction, the user goal and the persona description.
Relation to Prior Work
Three lines of work motivate the proposal. Zhou et al. (Mind the Sim2Real Gap) ran the human study that established the "easy mode." They found that simulators accommodate agent mistakes and front-load details, and that a simulator model's general capability does not track its fidelity. They report pass rates and a fidelity index, not task-level latent parameters, so they cannot say which tasks carry the gap. Persona Policies (Beyond Cooperative Simulators) evolves programs that make simulated users more human-like on τ²-bench. Annotators judged those users human 80.4% of the time. That work improves the simulator. We instead measure what bias remains under any simulator, Persona Policies included. ConvApparel (ConvApparel) shows that matching dialogue statistics does not guarantee correct behavior when the agent changes. That result constrains us: matching distributions is not enough, so our validation must be counterfactual across agents.
Two lines supply the method. Truong et al. (Reliable and Efficient Amortized Model-based Evaluation) predict IRT difficulty from frozen embeddings of single-turn question text with a linear model. Agent Psychometrics predicts the difficulty of agentic coding tasks from issue text and repository artifacts, adds LLM-as-judge features, and decomposes agent skill into model and scaffold components. Neither involves a user in the loop. We reuse their embedding-plus-judge predictor, but point it at a different target: the difference between human and simulated difficulty, rather than difficulty itself.
One line constrains the design. Chen et al. (LLMs Struggle to Measure What Distinguishes Students) find that LLMs predicting discrimination directly from text reach a Spearman correlation of only 0.152, and that synthetic persona responses reach 0.241. IDGen reports a trained discrimination predictor for single-turn prompts, but checks it only through an aggregate discrimination index, not per-item correlation. We take Chen et al.'s negative result seriously and do not attempt to predict discrimination from text. We estimate discrimination compression empirically and treat it as a reported limitation of simulator-based evaluation.
Research Program
The work has three parts.
The crossed study. On τ-bench retail and airline (115 and 50 tasks, τ-bench) and on τ²-bench telecom (τ²-Bench), we run six agents spanning a wide range of skill. Each agent faces three user sources: recruited human participants who play the scripted user goal, a default prompted simulator, and the Persona Policies simulator. We use several simulator base models, because Zhou et al. show that simulator fidelity varies by model family. Every agent meets every task under every simulator, with four repeats each. On the human side, each agent meets each task once with a different participant, and participants are reused across tasks so their personal severity can be estimated. If Zhou et al.'s human traces are available, they supply additional human-side data. The human side totals roughly a thousand conversations, which is the same order as Zhou et al.'s study.
Fitting and diagnosis. We fit the model above with Bayesian hierarchical estimation. Success is binary, as defined by τ-bench's database-state check. We also fit nested variants: one with only a global leniency term, one with task-specific shifts, and one that additionally lets discrimination vary by source. Comparing these says how much of the gap is uniform, how much is task-specific, and how much is compression.
The shift predictor. We train a predictor that maps each task's text to the estimated shift $\delta_i^{(s)}$. The input is the user instruction, the hidden goal, the persona, and the domain policy the agent must follow. The model is ridge regression on frozen embeddings plus LLM-judge features that ask, for example, whether the task requires the user to push back or to withhold information. Evaluation uses held-out tasks and a held-out domain.
The practical output is a corrected simulator score: each agent's skill estimated from simulator runs alone, with predicted shifts subtracted. We also test a hybrid in which a small number of human rollouts on the most informative tasks anchor the correction.
Baselines
Raw simulator pass rate. The naive reference is what τ-bench reports today: an agent's success rate against the default simulator, taken as its ability. The question is how well it reproduces human-based agent ranking and per-task human outcomes. Zhou et al. report the aggregate inflation (77.8% versus 63.6%). For per-task or per-agent agreement between simulator and human outcomes, no reported result was found on Lacuna. We compute it ourselves on identical tasks and agents.
Global offset correction. This is IRT on simulator data with a single leniency term $u_s$, calibrated on the same human anchor tasks our method uses. It is the strongest cheap competitor. If the gap is mostly uniform, it will match us. Our method beats it if, on held-out tasks, it gives better log-likelihood and calibration (Brier score) for human outcomes, and a higher Kendall rank correlation with the human-derived agent skills.
Persona Policies with no correction. This is the strongest existing method for closing the gap (Beyond Cooperative Simulators). It runs with the same agents, tasks and number of repeats, using the generator released or reproduced per that paper. Its reported numbers are an 80.4% human-likeness rating and +17% agent robustness. Neither is a per-task agreement measure, so no directly comparable figure exists on Lacuna. Our correction applied on top of Persona Policies should still improve agreement with humans. If it does not, the remaining gap is not task-specific.
Text-based difficulty predictor. The Truong et al. / Agent Psychometrics approach predicts human difficulty $b_i$ directly from text, with no simulator runs. The closest reported number is an AUC-ROC of 0.845 on SWE-bench Verified (Agent Psychometrics). That is a coding benchmark with no user, so it is not comparable. We run the same feature set and tuning budget on our targets.
Ablation of the key ingredient. Replacing predicted shifts with zero recovers the global offset baseline. Replacing them with zero-shot LLM-judge estimates isolates the contribution of the trained text-to-shift mapping. All predictors get the same embeddings, the same judge model and the same cross-validated regularization search.
Why not just…?
Why not just build a better simulator? This is the real threat. If Persona Policies or a fine-tuned simulator drove the task-specific shifts to near zero, the correction would be unnecessary. The crossed study answers this directly, because it estimates the shifts separately under Persona Policies. Note that "rated human 80.4% of the time" measures style. ConvApparel shows that style realism and behavioral correctness can come apart. Even in the best case, a better simulator does not tell you how much bias remains. Our measurement does.
Why not just calibrate a single correction factor from a few human runs? If the gap were uniform, this would be enough, and we include it as a baseline. Zhou et al.'s qualitative finding argues against uniformity: simulators specifically smooth over agent errors and volunteer details early. That should matter most for tasks that depend on eliciting information or enforcing policy against a pushy user. The nested model comparison settles the question. If task-specific shifts do not improve held-out prediction, we will report that the gap is uniform, which would itself be a useful, simplifying result.
Why not just compare agent rankings under the simulator and ignore absolute scores? A uniform easy mode would be harmless for rankings. However, compressed discrimination means simulators bunch agents together and amplify noise in their ordering, and task-specific shifts can favor agents whose weakness is in exactly the tasks the simulator makes easy. We report rank correlation with human-derived skills as a primary metric, so if rankings are preserved, this check will show it.
Why not just ask an LLM which tasks real users would make harder? Chen et al. show that zero-shot LLMs predict a closely related property, discrimination, poorly. Agent Psychometrics finds that judge features help most when combined with embeddings. We include the zero-shot judge as an ablation. If it matches the trained predictor, the cheaper method wins, and we will say so.
Proposed Experiments and Decisive Checks
Is the gap heterogeneous across tasks? On τ-bench retail and airline, we test whether task-specific shifts improve held-out human-outcome log-likelihood over the global offset model, under both the default and Persona Policies simulators.
Is the gap predictable from text? We measure the Spearman correlation between predicted and estimated shifts on held-out tasks, and under transfer from retail to airline and to τ²-bench telecom. As a reference point, Chen et al.'s 0.241 for discrimination is the bar a useful predictor should clearly exceed.
Does correction recover the human verdict? With a held-out agent, we compare the Kendall rank correlation between simulator-derived and human-derived agent skills, and the Brier score on held-out human conversations, across raw simulator, global offset, Persona Policies, and our corrected estimates. We also measure how many human anchor tasks the hybrid needs.
How much discrimination is lost? We report the ratio of simulator to human discrimination per task. We expect this ratio to fall below one and not to be repaired by correction, which would define the limit of what simulator-based evaluation can measure.
If the first check fails, the proposal reduces to a careful demonstration that the gap is a constant, which is also worth publishing. If the second fails, the human-anchored hybrid remains as a cheaper route to human-equivalent agent rankings.
Workshop authors to connect with
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents: Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques
- LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment: Han Chen, Ming Li, Chenguang Wang, Yijun Liang, Dawei Zhou, Hong Jiao, Tianyi Zhou
References
- Mind the Sim2Real Gap in User Simulation for Agentic Tasks
- Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
- ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
- Reliable and Efficient Amortized Model-based Evaluation
- Agent Psychometrics: Task-Level Performance Prediction in Agentic Coding Benchmarks
- LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
- IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
- {$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment