Turing Rewards That Know What the Speaker Saw

Language-model user simulators are now how many conversational agents get trained and scored. A recent idea trains them by imitation against a judge: a second model tries to tell the simulator's replies apart from real human replies, and the simulator is rewarded for fooling it. In many of the settings that matter, though, each speaker sees something the other does not, such as their own screen, their own cards, or their own view of a shared scene. A judge that reads only the transcript can be fooled by a reply that is fluent and human-sounding but false about the speaker's own private situation. A judge that reads the whole world state has the opposite problem: it punishes the simulator for not using information it could never have seen. We propose a third design. The judge sees exactly what the imitated human saw and nothing more, and so asks whether a given person, looking at this particular view, could have said this.

The claim is that this choice of what the judge sees is the main design decision when doing adversarial imitation of dialogue under partial observability. Getting it right should yield simulators that are both human-like and faithful to their own hidden state. That pairing is what makes a simulator usable as an evaluator. If the simulator is good, an agent's success rate measured against it should predict that agent's success rate against real people. We test this in a collaborative reference game where human-human dialogues were recorded together with each player's private view, so the right conditioning information is actually available.

Core Claim

Adversarial imitation learning trains a policy (the generator) to make its behavior indistinguishable, to a learned classifier (the discriminator), from expert demonstrations. The classifier's output is then used as a reward (Generative Adversarial Imitation Learning). Write $o^i$ for the private observation of the speaker being imitated, $h_t$ for the shared dialogue history at turn $t$, and $u_t$ for the candidate utterance. We train a discriminator $D_\phi(o^i, h_t, u_t)$ and give the simulator the turn reward

$$r_t = \log D_\phi(o^i, h_t, u_t) - \log\big(1 - D_\phi(o^i, h_t, u_t)\big).$$

By the standard occupancy-matching argument behind this family of methods, the fixed point is a simulator whose conditional distribution $p(u \mid o^i, h)$ matches the human one. This is the distribution a real person with that information set produces.

The two obvious alternatives aim at different targets. A judge that sees only the transcript aims at the human distribution averaged over all possible private views. It can be satisfied by replies that fit some view, not necessarily the speaker's own. A judge that sees the global state $s$, including the partner's view, aims at $p(u \mid s, h)$. The simulator cannot represent that distribution because it never observes $s$, so part of the reward is noise the policy cannot influence.

The research file flags this tension in its first candidate gap but leaves it unresolved. We make it the object of study. Our prediction is that information-set conditioning beats both alternatives on private-state faithfulness and on evaluator fidelity, while matching the transcript-only judge on human-likeness.

Relation to Prior Work

The method comes from two lines of work. The seed paper Learning User Simulators with Turing Rewards introduced Turing-RL. It trains a simulator against an LLM judge that sees the user's history and shows this beats likelihood training and similarity rewards on chat and Reddit data. Its judge conditions only on visible text, and its domains have no hidden state, so the failure we target cannot arise there. Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning used a discriminator over discrete dialogue acts as a dense turn-level reward, reaching 87.52% task success against 81.24% for plain A2C. Its setting has a fully observed slot tracker and a rule-based user. Multi-Agent Generative Adversarial Imitation Learning showed that the structure of the discriminator (centralized, decentralized, or zero-sum) matters in multi-agent imitation. It worked only on 2D particle environments with vector communication. We carry its lesson, that discriminator structure is an inductive bias, into natural language, and we ask specifically which observations the discriminator should be given.

The problem is motivated by work showing that simulators and agents break under hidden state. τ²-Bench formalizes dual-control support dialogue as a decentralized partially observable problem. It reports that tool-constrained user simulators still err 16% of the time and prompt-only ones 40%, so simulator faithfulness to private state is a measured bottleneck rather than a hypothetical one. τ-bench shows that agent reliability collapses over repeated trials (pass^8 below 25%), which makes reliable simulated evaluation more valuable. The seed paper Moral Hazard in Multi-Agent Language Models shows that language agents in hidden-action games often fail to turn private information into useful messages. Training or auditing such agents requires partners that behave like people who hold specific private facts.

The testbed and the strongest existing partner model come from the reference-game literature. Reference-Centric Models for Grounded Collaborative Dialogue built a modular supervised ONECOMMON agent with pragmatic re-ranking. It reaches 46.0% success with human partners versus 30.6% for the prior model, against roughly 66% for human pairs. Modeling Perspective-Dependent Ambiguity in Collaborative Dialogue shows that egocentric agents, which assume the partner sees what they see, fail. It reaches 85.2% in symbolic self-play but reports trouble turning plans into language. These papers build agents. We build a simulator of the human partner, and we use their published human-evaluation numbers as ground truth for checking simulator fidelity. The seed paper No One Wins in Nuclear War offers private-channel games with replay-verifiable hidden state, which is a natural later target once the method is validated where human data exist.

Research Program

The main testbed is ONECOMMON. Each human player sees seven dots that partly overlap with the partner's view, and the pair must agree on a shared dot through free-form dialogue. Each recorded utterance therefore comes with the speaker's private view, which supplies the $o^i$ that our discriminator needs. The private-state facts are checkable: dot size, color, and position are known, so we can tell whether an utterance is literally true of the speaker's own view.

The simulator is an open-weight language model of about 7B parameters, conditioned on a text serialization of its own dot view. It is first trained with supervised fine-tuning on human turns and then trained further with RLOO-style policy gradients on $r_t$. Rollouts are interactive multi-turn dialogues, not single-reply continuations. The simulator plays opposite a frozen partner model sampled from a small pool. This exposes it to states its training transcripts never visited, which is where adversarial imitation is supposed to beat copying.

The discriminator is a smaller fine-tuned classifier, not a prompted judge, so that its inputs can be controlled exactly. It is trained to separate human turns from simulator turns under three conditioning regimes:

  • transcript only;
  • the speaker's own view (the proposal);
  • the full joint state.

We refresh the discriminator periodically and keep a replay buffer of past simulator turns, which limits the usual instability of adversarial training.

Two analyses accompany training. The first is a diagnostic check of the claim that the global-state judge injects noise the policy cannot affect. We measure how much of the reward variance is explained by the partner's view after conditioning on the speaker's own view. Second, we look at which human regularities the information-set judge picks up that a fact-checker would not, for example asking more clarification questions when the overlap between views is small, or which salient dots people mention first.

Baselines

The do-nothing reference is a prompted open-weight model of the same size. It is given the dot view in text and asked to play a human, with no training. It shows how much cooperative bias and view-hallucination the untrained base model has. We expect unrealistically smooth dialogues and frequent claims about dots the speaker does not have.

The supervised baseline is the same model fine-tuned on human ONECOMMON turns conditioned on the speaker's view. This is behavior cloning with the correct information set. It is the strongest cheap competitor, because it already sees what the speaker saw, and any gain from our method has to come from the interactive, discriminator-based training stage. It receives the same data. Its compute budget is set equal to supervised training plus RL for our method, spent on more epochs and hyperparameter search. Beating it means higher private-state faithfulness and better evaluator fidelity on rollouts, not just matching its held-out likelihood.

The closest prior method is Turing-RL as described in the seed paper, implemented with a transcript-only discriminator. It uses the same generator, data, number of RL steps, and discriminator capacity, and receives the same hyperparameter sweep as our method. The file contains no reported Turing-RL number on ONECOMMON; the abstract reports only that it beats likelihood and similarity baselines on chat and Reddit. Beating it means equal or better human-likeness together with a clearly lower rate of utterances that are false about the speaker's own view.

The strongest existing partner model is the FULL+PRAG system of Reference-Centric Models for Grounded Collaborative Dialogue, which reports 46.0% success with human partners and 83.3% corpus choice accuracy. It was built as an agent, not a human simulator, but it is the best existing model of ONECOMMON behavior and the natural candidate for a simulated partner. Beating it as a simulator means its pairings with test agents are further from human-measured success than ours.

The key ablation swaps the discriminator's conditioning to the full joint state, with everything else held fixed. If it matches our method, our information-asymmetry argument is wrong. A second ablation gives the discriminator the speaker's view but shuffles views across examples. This checks that gains come from the right view and not from extra input tokens.

Why not just…?

Why not just fine-tune on human turns with the speaker's view in the prompt? This is the baseline most likely to be underrated, because it already has the right information set. What it lacks is any signal about the simulator's own interactive mistakes. Once a simulated dialogue drifts away from the training transcripts, nothing corrects it. MAGAIL reports behavior cloning failing in cooperative particle tasks where adversarial imitation reached near-expert reward, and the Turing-RL seed paper reports likelihood training losing to discriminative rewards. Neither result is on grounded dialogue, so our supervised-versus-adversarial comparison on interactive rollouts settles this directly. If the supervised model matches us, the conclusion is that conditioning matters and adversarial training does not.

Why not just add a fact-checker reward to transcript-only Turing-RL? This is the most serious threat. ONECOMMON facts are machine-checkable, so a reference resolver could penalize false claims, and the transcript judge would handle human-likeness. We run this combination as a full baseline, using a resolver of the kind in Fried et al. as the checker. Our reason to expect a gap is that a checker rewards literal truth, while human behavior depends on the private view in ways that are not true-or-false. Examples are which dots people choose to mention, when they hedge, and how much they ask when overlap is small. These dependencies drive evaluator fidelity. The checker also has to be rebuilt for each domain, while the conditioned judge only needs paired logs. If the combined baseline matches us on both faithfulness and fidelity, we will report that the factored design suffices whenever a checker exists, and limit our claim to domains without one.

Why not just prompt a frontier model with the view? The file's evidence argues against zero-shot competence under partial observability. τ²-Bench reports GPT-4o falling from 74% in single-control retail to 34% in dual-control telecom, and the ONECOMMON papers document egocentric failures. Those results concern agents, not simulators, and the file has no reported number for frontier-model ONECOMMON simulators. We therefore include a prompted frontier model as an extra reference on the faithfulness and fidelity metrics rather than assume it fails.

Proposed Experiments

We report three metrics on held-out ONECOMMON scenarios.

Human-likeness is measured by human raters who see a scenario, the speaker's view, and two candidate turns, one real and one simulated. The score is the rate at which raters cannot pick the human turn. Raters see the view, so a fluent but false reply is penalized.

Private-state faithfulness is the fraction of referring expressions that resolve to dots actually in the simulator's own view. It is checked automatically and audited by hand on a sample.

Cooperative bias is measured by comparing simulator-plus-agent success, dialogue length, and clarification rates against the human-human reference of about 66% success. A simulator that makes agents succeed more often than human partners would is too easy.

The decisive check is evaluator fidelity. We take a panel of agents of varying quality: the older ONECOMMON baseline, FULL+PRAG, a supervised LLM agent, and a self-play-tuned LLM agent. We measure each one's success with real crowdworkers in a modest human study, anchored on Fried et al.'s reported 46.0% for FULL+PRAG and 30.6% for the prior model. We then measure each agent's success against each candidate simulator. A good simulator should reproduce the human ranking of agents (Spearman correlation) and the absolute gaps (mean absolute error in success points). Our prediction is that the information-set judge is best on both. The transcript-only judge should overestimate success, because its replies are agreeable but sometimes false about the view. The global-state judge should train less stably and land between the two.

As a secondary check of generality, we will apply the trained recipe to private-channel episodes from the WOPR engine and to the Dialogue Moral Hazard Game. Neither has human logs on the file, so there we report only private-state faithfulness and qualitative behavior, not human-likeness claims.

Workshop authors to connect with