When a customer-service agent built on a large language model fails a conversation, two very different things may have gone wrong. The task may be hard: a tangled refund policy, or a seven-step troubleshooting sequence. Or the user may be hard: impatient, vague, reluctant to share an order number. Today's interactive benchmarks report one success rate per task. They pair an agent with a language model playing the user and check whether the final database state is correct, so the two sources of failure cannot be told apart. This proposal treats every conversation as a test item defined by a task and a simulated user personality together. We fit a psychometric model across many agents. Psychometrics here means the statistics of test design, where each item gets a difficulty and each test-taker an ability. The model asks how much of an item's difficulty belongs to the task, how much to the persona, and how much exists only in their combination. We then try to predict that difficulty for new task–persona pairs from one cheap pilot conversation, without running dozens of expensive ones.
The two research directions meet here. Work on estimating item difficulty has shown that the process of solving an item carries more signal than the item's text. Examples include the reasoning episodes inside a model's chain of thought (Epi2Diff) and the steps of a coding agent (PTA-IRT). Work on user simulation now produces users realistic enough that the user has become a real source of difficulty (PPol). Nobody has measured that source. Without such a measurement, evaluations can mistake a robust agent for a capable one, and costly repeated runs get spent on conversations that tell us nothing.
Core Claim
Our claim has two parts, and each can fail on its own. First, success in tool-using customer conversations is well described by a cross-classified item response model. An item response model is a logistic model in which the chance a test-taker succeeds depends on their ability minus the item's difficulty. In our version, the item's difficulty splits into a task part, a persona part, and a low-rank interaction term. Agents also differ in how much user friction hurts them, which is separate from their raw skill. Second, the fitted difficulty of an unseen task–persona pair can be predicted better from a short pilot dialogue than from the task and persona descriptions alone. The pilot dialogue is labeled into conversational episodes such as clarification requests, repairs after misunderstanding, user refusals, and policy checks.
Formally, for agent $j$, task $t$ and persona $p$, the probability of success is
$$P(y_{jtp}=1)=\sigma\big(a_{t}\,[\theta_j - b_t - c_p - \mathbf{u}_t^\top \mathbf{v}_p - \gamma_j c_p]\big),$$
where $\sigma$ is the logistic function, $\theta_j$ is agent skill, $a_t$ is how sharply task $t$ separates strong from weak agents, $b_t$ is task difficulty, and $c_p$ is persona friction. The low-rank term $\mathbf{u}_t^\top \mathbf{v}_p$, of rank 1–3, captures task–persona combinations that are hard together, for example a reluctant user on a task that needs an account identifier. The coefficient $\gamma_j$ captures agent-specific sensitivity to friction. If $\mathbf{u}_t^\top\mathbf{v}_p$ and $\gamma_j$ do not improve held-out fit, the decomposition reduces to averaging, and we will say so.
Relation to Prior Work
Three groups of papers shape the project, each in a different role.
The first group motivates it by showing the gap. Sedoc & Ungar treated chatbot prompts as test items and selected the most informative ones to halve human evaluation cost. Their items were single-turn and their judges were human crowd workers, and they left multi-turn conversation open. τ²-Bench gives the user their own tools in a shared world. It shows a roughly 20% "coordination tax" when the agent must guide the user rather than act directly, but it reports this as an aggregate drop, not a per-item quantity. TRACER splits runtime failure risk on τ²-bench into an agent-side gap and a user-coordination gap. That is the closest existing idea. TRACER, however, is an early-warning detector on individual trajectories and does not calibrate stable difficulty parameters for tasks or personas. We use its signals as a baseline and as candidate features.
The second group supplies the method. Agent Psychometrics splits coding-agent ability additively into model and scaffold, and predicts task difficulty from task artifacts. We apply the same additive logic on the item side, splitting task from persona, and borrow its artifact regression as a baseline. PTA-IRT shows that trajectory summaries sharpen item response estimates enough to rank agents from a 10% subset. Epi2Diff converts monologue reasoning traces into sequences of cognitive episodes that predict human item difficulty. We carry the episode idea from a single model's private reasoning into a two-party dialogue, where the episodes that matter involve the user. PPol supplies the persona population: evolved programs that inject impatience, ambiguity and reluctance while keeping the task goal intact.
The third group constrains the design. Li et al. found that asking language models to rate difficulty, or to role-play weak students, correlates poorly with human difficulty, with mean Spearman correlation 0.28. Their predictions also cluster around the mean. This rules out using an LLM judge as our main estimator. It also warns that our ground truth is difficulty for a population of agents, not for humans. We keep that distinction explicit and do not claim to predict human difficulty. Zotos et al. show that token-level uncertainty tracks difficulty only when the model is right, and TRACER notes that averaging surprisal washes out sparse breakdowns. Both argue for structured process features over raw confidence.
Research Program
We build a calibration set on the τ²-bench Retail, Airline and Telecom domains. Telecom has 114 tasks; the retail and airline task pools follow τ-bench, with 115 and 50 tasks.
Users come from a fixed population of about 20 PPol-generated personas plus the default cooperative simulator, which serves as a zero-friction anchor. The test-takers are about a dozen agents spanning open and closed models and two scaffolds. Running the full agent × task × persona grid with repeats is unaffordable. We therefore use a sparse, balanced incomplete design: each agent sees a random subset of task–persona pairs, with enough overlap that all parameters are identified. The item response model handles the missing cells. Repeats are concentrated on a held-out reference subset, which we use to estimate run-to-run noise from the stochastic simulator.
The second component is a conversational episode labeler. Following Epi2Diff, a language model labels each turn of an agent–user dialogue with a small taxonomy. The draft categories are agent information request, user disclosure, user withholding or deflection, misunderstanding, repair, policy lookup, tool call before required arguments are gathered, instruction to user, user action error, and premature closure. We validate the taxonomy by having two annotators label 200 dialogues and report agreement. We then compute compact process features: episode counts, transition frequencies (for example misunderstanding followed by repair), and the turn at which required information is first obtained.
The third component is a predictor. For a new task–persona pair, we run one pilot dialogue with a single cheap fixed agent, extract episode features, and regress onto the fitted $b_t + c_p + \mathbf{u}_t^\top\mathbf{v}_p$. The main generalization tests hold out whole tasks, whole personas, and both at once.
Baselines
The naive reference is marginal averaging. Task difficulty is one minus the task's mean success rate across all agents and personas, and persona friction is the drop in mean success relative to the cooperative simulator. No model is fit. This is the "do nothing psychometric" option. Our cross-classified model must beat it in held-out log-likelihood of individual dialogue outcomes and in held-out agent-ranking accuracy, on identical data.
The second baseline is task-only item response theory in the style of Sedoc & Ungar and PTA-IRT. It fits $b_t$ and $a_t$ while ignoring which persona was used, and so treats persona variation as noise. Comparing against it shows whether modeling the user changes conclusions, such as agent rankings or which tasks look informative. It also isolates the value of persona parameters.
The strongest existing predictor of difficulty from artifacts is the Agent Psychometrics recipe: ridge regression over embeddings of the task instruction and persona program, plus LLM-judge features. It reports task-difficulty AUC-ROC of 0.845 on SWE-bench Verified (Agent Psychometrics). We found no reported result on Lacuna for difficulty prediction on τ²-bench. We will rerun the recipe on our data with the same train/test splits and the same tuning budget, by cross-validated ridge penalty, as our predictor. Beating it means higher held-out Spearman correlation with fitted difficulty under task-held-out and persona-held-out splits, with confidence intervals from bootstrap over tasks that do not overlap.
The strongest process baseline is TRACER's trajectory features (tail-risk surprisal, repetition, agent coherence gap, user coordination gap). They are computed on the same single pilot dialogue and fed to the same regressor. TRACER reports detecting 97.3% of failed Retail tasks within the first 20% of a trajectory and up to 37.1% AUROC improvement for failure prediction (TRACER). Its target is the outcome of one run, so it is not directly comparable to our population difficulty, which is why we refit its features to our target.
Two ablations isolate the key ingredient. In the first, episode labels are shuffled within dialogue while counts are kept, which tests whether the order of episodes matters. In the second, an Epi2Diff-style monologue condition labels only the agent's own reasoning on the task description, with no user turns. This tests whether the user's side of the dialogue carries the signal.
Why not just…?
Why not just average pass rates per task and per persona? This is the most serious threat. If difficulty really is additive, and agents respond to friction uniformly, then simple averages recover the same quantities and the psychometric model adds only ceremony. We do not argue around this. The first decisive check compares the additive model against the full model on held-out cells. If the interaction rank and $\gamma_j$ add nothing, we report that persona friction behaves as a constant tax. That is itself useful, because it would mean benchmark designers can score a persona once and reuse the score. The prediction component would still stand.
Why not just predict difficulty from the task and persona text, as Agent Psychometrics does for coding? That approach works well on SWE-bench. In conversation, however, much of the difficulty arises only once a specific user meets a specific requirement. A reluctant persona is harmless on a task that needs no private information. Text regression can learn this only from many labeled combinations. Our held-out-persona and held-out-both splits test directly whether a pilot dialogue generalizes where text features do not.
Why not just ask a strong model how hard each conversation will be? Li et al. report mean Spearman correlation 0.28 for this kind of observer judgment on static items, with predictions clustered around the mean (Li et al.). We still include an LLM judge, given task and persona, as a cheap baseline, because conversational settings may differ. We expect it to sit below both regressors.
Why not just run pass^k on everything? Repeated runs are the honest gold standard, and we use them to build ground truth. Their cost is high: τ-bench's $pass^8$ needs eight full dialogues per task (τ-bench). They also spend the same budget on uninformative items as on informative ones. The adaptive-selection experiment below measures how much of that budget calibrated items save.
Proposed Experiments
The first experiment tests decomposition. We fit the additive, interaction, and full models and compare held-out log-likelihood on 20% of cells. We report the share of difficulty variance attributable to task, persona, and interaction in each domain. We expect Telecom, the dual-control domain, to show the largest persona and interaction shares, given its documented coordination tax: GPT-4o drops from 74% in single-control Retail to 34% in Telecom (τ²-Bench). If Telecom shows no larger persona share than Retail, that expectation is wrong. A secondary check is whether agent rankings under the cooperative simulator differ from rankings under the persona population. Rank changes would mean current leaderboards measure cooperation-conditional skill.
The second experiment tests prediction. The episode-feature predictor, TRACER features, artifact regression, LLM judge and monologue ablation are compared under task-held-out, persona-held-out, and both-held-out splits. The decisive result is the both-held-out split. If one pilot dialogue does not beat artifact regression there, the process-trace claim fails for conversation.
The third experiment tests efficient evaluation. We estimate a new agent's $\theta_j$ by choosing task–persona pairs with the highest Fisher information under predicted parameters, as PTA-IRT does for coding. Fisher information here measures how much an item's outcome narrows the estimate of the agent's ability. We compare against random selection and task-only selection at 5%, 10% and 20% budgets. The metric is Kendall's τ against the full-grid ranking. PTA-IRT reaches 0.888 at a 10% budget on SWE-bench (PTA-IRT), a reference point from a different benchmark rather than a target to beat.
The main limitation is that the personas are simulated. Their difficulty is difficulty against simulated humans that PPol judges rated human-like 80.4% of the time, not difficulty against real users. A small study is the natural follow-up: crowd workers act as users under assigned behavioral instructions on about 40 tasks, and we check whether fitted persona friction ranks the same way.
Workshop authors to connect with
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents: Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction: Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou
References
- Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
- Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
- Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
- Item Response Theory for Efficient Human Evaluation of Chatbots
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning
- Agent Psychometrics: Task-Level Performance Prediction in Agentic Coding Benchmarks
- Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
- Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty?
- {$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains