Preference Optimization That Does Not Trust the User Simulator

Conversational agents that guide real people through multi-step tasks, such as a support agent walking a customer through phone settings, are increasingly trained on conversations with a simulated user played by another language model. The simulated user is cheap, but it is a poor stand-in for a person. It makes things up, it is too agreeable, and it often ends the call early. We propose a training method in which an agent action is preferred only when it leads to success across a varied group of simulated users, and only in rollouts where the environment confirms the simulated user did what it claimed. Rollouts where the simulator lied or hallucinated are dropped before any preference is formed. The goal is an agent whose learned habits, such as checking whether the user actually toggled a setting before moving on, carry over to real humans. A naively trained agent instead learns habits that only work on a gullible simulator.

The training algorithm is direct preference optimization (DPO). DPO fine-tunes a language model straight from pairs of "better" and "worse" responses, with no separate reward model and no reinforcement-learning loop. Our claim is about where those pairs come from. In multi-turn training, the pair labels come from simulator rollouts. When the simulator is wrong 16 to 40 percent of the time, as reported by $\tau^2$-Bench and $\tau$-bench, the labels systematically reward shortcuts that exploit the simulator. This is the conversational analogue of the shortcut problem in robotics and vision transfer from simulation to reality. We argue it is fixed at the labeling stage, not by changing the loss.

Core Claim

Existing multi-turn preference methods treat the simulator's outcome as ground truth. Direct Multi-Turn Preference Optimization for Language Agents (DMPO) fixes the math of DPO across turns. It replaces the per-step constraint with a constraint on how often state-action pairs are visited, so the normalizing term cancels, and it adds length normalization. It is evaluated only in clean programmatic environments (WebShop, ScienceWorld, ALFWorld). Learning to Clarify (ACT) samples agent responses, rolls them out against one LLM user simulator, and scores outcomes with task-specific checks. It reports a jump in AmbigSQL post-clarification execution match from 13.9% to 38.1% with 50 examples. It never asks whether those gains depend on the simulator being cooperative. FSPO shows that synthetic preference data can transfer to real Prolific users, but only for single-turn generation.

None of these separates two kinds of success. In one, the agent's action caused the success. In the other, the simulator caused it, for example by falsely reporting a step as done or by volunteering information a real user would withhold.

Our hypothesis concerns dual-control settings, where the user holds private tools the agent cannot see. In these settings, a large share of simulator-derived preference pairs either flip or become uninformative once two changes are made. First, outcomes are averaged over an ensemble of differently behaving simulators. Second, rollouts in which the simulator's words contradict its own logged tool state are discarded. We expect a model trained on the cleaned pairs to show a smaller drop from training simulator to human users than one trained with DMPO or ACT-style labels on the same compute. If the flip rate turns out to be small, the idea is wrong and we will say so. That outcome is our main kill criterion.

Research Program

The setting is the telecom domain of $\tau^2$-Bench. Both the agent and the user act on a shared environment. The agent has billing and line tools; the user has a mocked phone with Wi-Fi, airplane mode and SIM controls. Success means the final database state matches the goal. This domain is the right testbed for three reasons. Its tool-constrained user simulator logs the user's actual tool calls, which gives us a way to check the simulator's claims. It shows the "coordination tax," a roughly 20% success drop when the agent must guide a user instead of acting alone. It also shows the long-horizon failure we care about, with success near zero once more than seven actions are required.

Pair construction works by branching. We take a partial conversation $h_t$ produced by the current policy and sample $K$ candidate agent turns. We then continue each candidate against $M$ simulated users drawn from an ensemble. The ensemble varies the underlying LLM, the persona, and the cooperativeness. It includes users who misreport, who answer clarifying questions vaguely, and who forget a constraint they stated early in the call.

Each rollout is audited. If the simulated user asserts a state change ("I turned airplane mode off") that its tool log does not contain, or the dialogue ends before the task's terminal condition, the rollout is marked as simulator-faulty and excluded. The score $\bar{s}(a)$ of a candidate turn $a$ is its mean final-state success over the remaining clean rollouts. We form a pair $(a_w, a_l)$ only when the difference in clean scores exceeds a margin and each candidate has at least a minimum number of clean rollouts.

Training uses DMPO's length-normalized multi-turn loss applied to the conversation suffix from the branch point, with each pair weighted by its estimated reliability:

$$\mathcal{L} = -\,\mathbb{E}\Big[\, w \,\log \sigma\big(\beta\,[\,\tilde{\ell}_\theta(a_w \mid h_t) - \tilde{\ell}_\theta(a_l \mid h_t)\,]\big)\Big].$$

Here $\tilde{\ell}_\theta$ is the length-normalized log-ratio of the policy to the frozen reference model over the agent's tokens from the branch onward. $\beta$ is the usual strength of the penalty for drifting from the reference model. $w$ is the fraction of clean rollouts behind the pair, scaled by the score gap.

Branching at early turns serves the long-dependency goal. It assigns credit to the specific turn where a constraint was dropped or a user claim went unverified, which DMPO's trajectory-level comparison does not do.

The first experiment needs no training. On rollouts from the base policy, we measure how often a preference built from a single simulator agrees with the cleaned ensemble preference. We break down disagreements by cause: false user reports, premature endings, and persona sensitivity. We also measure how disagreement depends on the depth of the branch point. This tells us whether the label problem is real before we spend compute on fine-tuning.

Baselines

The naive reference point is supervised fine-tuning on successful simulator conversations. The base model is an open 7–8B model of the kind used in DMPO and KL Penalty Control via Perturbation for Direct Preference Optimization, namely Mistral-7B and Llama-3-8B. We also report untrained prompting of the same base model. For an upper reference, $\tau^2$-Bench reports Claude 3.7 Sonnet as the best telecom model at 49% and GPT-4o at 34% pass@1. We found no reported result on Lacuna for any open-weights model fine-tuned on $\tau^2$-Bench telecom, so we will establish those numbers ourselves.

The strongest method-level competitor is DMPO. It is the only multi-turn DPO variant here with a principled fix for the cross-turn breakdown. It reports 0.651 reward on ScienceWorld unseen, against 0.629 for standard DPO, and 0.701 on WebShop, above GPT-4's 0.632. The second competitor is an ACT-style quasi-online DPO, which uses on-policy sampling scored by a single tool-constrained simulator. It is the existing recipe for simulator-driven multi-turn DPO. We also include turn-level standard DPO on concatenated histories.

For fairness, every method receives the same base model, the same task split, and the same total number of simulator-turn calls. We count each rollout in our ensemble against that budget, so our method sees fewer pairs. Every method also receives the same hyperparameter search budget, tuned on the training simulator only.

Three ablations isolate the key ingredient. The first keeps the ensemble but drops the faulty-rollout filter. The second keeps the filter but uses a single simulator. The third keeps both but forms whole-trajectory pairs instead of branch-point pairs.

Beating the competitors means the following on the human-played evaluation: higher pass^1 than DMPO and ACT-style training at matched budget, with confidence intervals that do not overlap across seeds, and a smaller drop in pass^1 from training simulator to humans. Higher pass^1 on the training simulator alone does not count. We expect, and would accept, that our method might score lower there.

Why not just…?

Why not just use the tool-constrained simulator and run DMPO? The tool-constrained simulator already cuts the error rate from 40% to 16%, so this is the obvious default. But 16% is a per-dialogue rate. Over a long conversation, and across the many rollouts needed per pair, a sizeable fraction of pairs can still rest on a faulty rollout. Premature termination also remains. The simulator-only-plus-DMPO arm is exactly our main baseline, and the training-free agreement study measures how much of its label set is contaminated.

Why not just randomize the simulator, the way robotics randomizes physics? This is the most serious threat. Diverse personas and noisy users may by themselves teach the agent to verify and re-ask, much as active noise injection in Adversarially Masking Synthetic to Mimic Real forces robust features. Randomization adds variety, but it does not remove labels that credit the agent for a simulator's false report. A dishonest simulator can still hand the "trusting" action a win. The ensemble-without-filter ablation is precisely randomization plus DMPO. If it matches the full method on humans, the contribution reduces to domain randomization and we will report that.

Why not just let a noise-robust DPO variant absorb the bad labels? KL Penalty Control via Perturbation adapts $\beta$ per pair and reports AlpacaEval 2 length-controlled gains for Mistral-7B from 26.8% to 35.6%. It infers noise from the model's own confidence, which works when noise is random. Simulator errors are systematic: the cooperative simulator consistently favors the same shortcut. A confident model will therefore treat those wrong pairs as clean. We run ε-DPO on the uncleaned pairs as a further baseline. If it closes the human gap, our filter is unnecessary.

Why not just fine-tune on real human transcripts? Human corpora such as JDDC and Ubuntu lack environment state, so success cannot be verified. They also lack the dual-control tools. JDDC further documents that maximum-likelihood training collapses toward a generic phrase in nearly a quarter of outputs. Human data is used here only for evaluation, where it matters most.

Proposed Experiments

Training uses the $\tau^2$-Bench telecom task generator. Evaluation has three tiers.

The first tier is the training simulator, which measures in-distribution performance.

The second tier is a held-out simulator family built on a different underlying LLM with unseen personas. It serves as a cheap proxy for the gap to reality.

The third tier is a human-played subset. Prolific participants, recruited as in FSPO, receive telecom scenarios and the same mocked phone interface, and their tool actions are logged. In every tier we report pass^1 and pass^k, using the reliability metric of $\tau$-bench, which reports that 60% pass^1 often falls below 25% pass^8. We also report success bucketed by the number of required actions, the rate at which the agent asks the user to confirm a state change, and response length. Length matters because Refining Text Generation for Realistic Conversational Recommendation found DPO-trained outputs grew longer and less diverse.

As a secondary check that the effect is not specific to telecom, we retrain ACT on AmbigSQL against a simulator that sometimes answers clarifying questions vaguely. We then test whether the cleaned-label version keeps its execution-match advantage where the original's 38.1% may not hold.

Three outcomes are decisive. First, the training-free study must show that a non-trivial share of single-simulator preference pairs flip under cleaning, with that share concentrated at early branch points. Second, on human users, the full method must beat both DMPO and the randomization-only ablation at matched simulator budget. Third, the method's ranking on the held-out simulator family must predict its ranking on humans. If the third holds, the cheap proxy can stand in for human studies in future work. If it fails, that negative result is itself worth publishing.