Conversational agents are increasingly trained and tested against simulated users: language models asked to play the human side of a dialogue. These simulators are too nice. They state their whole request up front, stay polite, and slide back into assistant behavior after a few turns. Agents that look strong against them look much weaker against people. In Flipping the Dialogue, GPT-4o's task score falls from 74.6% to 57.4% when the prompted simulator is swapped for a model trained to talk like real users. A natural fix is to let the simulator reason privately before each turn about what it feels, what it has already revealed, and how patient it still is. Chain-of-thought prompting, meaning generating intermediate reasoning text before the answer, is the obvious tool. But in a long dialogue that reasoning piles up in the model's input window, and the growing transcript is exactly what makes models forget their initial persona instructions.
We propose a user simulator whose private reasoning is a small, fixed-size "user state" note. The simulator rewrites the note from scratch before every utterance and then drops the earlier versions. It is never appended. The note carries the recurrent state a real person has (mood, patience, goal, what has been disclosed, what is being held back) at constant context cost. We train it from hindsight: a labeler that reads a complete real conversation writes down what the user's state must have been at each turn. The claim is that overwrite-style user deliberation keeps the realism benefits of reasoning while avoiding the long-horizon drift that appended reasoning makes worse. This should produce simulators that stay diverse and stay "in person" over dozens of turns.
Core Claim
Three findings in the literature pull against each other. Autoregressive + Chain of Thought = Recurrent argues that a transformer can only carry state that evolves step by step if it writes that state out as tokens and re-reads them. The same paper notes that this tape grows linearly and is costly. Measuring and Controlling Persona Drift traces persona loss to attention on the system prompt decaying each time new turns are appended; every model it tested drifted within 8–16 turns. Reflect, Not Reflex shows that an explicit inference step before responding makes dialogue replies much less generic. It does this only for the assistant, and only turn by turn.
Put these together and we get a prediction. Appended chain of thought for the user side should improve realism early in a conversation and then accelerate drift. A rewritten, bounded note should keep the early gain without the late collapse. What is new here is not "chain of thought for user simulators" in general. It is two things together. The first is the specific structure: a recurrent, overwritten, persona-anchored state. The second is the hindsight supervision that teaches a mid-sized model to fill that state with human-like content, which means withheld intent, accumulating frustration, and sparse traits that surface only when triggered, rather than assistant-like content.
Relation to Prior Work
Several papers motivate the problem. Flipping the Dialogue trains user language models on role-flipped real chats and shows the "assistant paradox": the better aligned a model is as an assistant, the worse it is as a user. Its UserLM generates each turn directly, with no deliberation step, and it was not examined over very long horizons. UserRL trains agents against prompted simulated users and itself notes that those users are rigid and revert to default behavior. CharacterBench explains why prompting fails for sparse traits, such as specialized knowledge or idiosyncratic habits, which stay dormant unless something triggers them. A per-turn state note is one place such triggers can be tracked explicitly.
Other papers supply our method. Know You First and Be You Better extracts implicit profiles (objective facts plus Big Five traits) from real logs. It rewards a simulator when a separate extractor can recover the profile from the simulated dialogue, a check called cycle consistency. We reuse both its LMSYS-USP profiles and its cycle-consistency check, applied per turn to our state note rather than once per dialogue. From Reflect we borrow the finding that smaller models need curated reasoning demonstrations rather than zero-shot reasoning prompts. This is why we supervise the note rather than simply prompting for it.
A third group constrains the design. Best Friends, Not Forever finds that self-managed, model-written JSON state does not rescue long-horizon companions: user-state recall sits at near-chance levels of 0.214–0.250 on four-option tests. This is the most direct threat to our idea, and we address it below. BRIDGE stabilizes role-players with continuous latent state and a pull back toward the initial persona anchor. That requires changing the architecture, and the state cannot be read as text. We keep a text state so it can be inspected and audited. PersonaForge and SAGE produce gradual information disclosure and grounded personas through prompted rules. We ask whether the same behavior can instead come from a learned state. Finally, Upfront Chain-of-Thought compresses reasoning for math problems. Our overwrite note is a dialogue-specific counterpart whose content is defined by the persona, not by the task.
Research Program
The simulator is an 8B-parameter model. At each user turn it receives four inputs: the persona profile, the most recent state note, the last $w$ dialogue turns, and the assistant's newest message. It first writes a new note under a fixed token budget $B$ (we sweep 64 to 256 tokens), then writes the utterance. Only the newest note is kept. Context cost per turn is therefore roughly $|profile| + B + w \cdot \bar{t}$, where $\bar{t}$ is the average turn length. This is constant in the dialogue's length, whereas appended reasoning grows linearly.
The note has a loose schema drawn from the user's side of the conversation. Its fields are current emotion, perceived progress of the assistant, remaining patience, goal (which may shift), facts already disclosed, facts deliberately withheld, and any sparse trait triggered in this turn. It is written in free text rather than as slot values. That choice follows the finding in MultiWOZ 2.2 that slot-based state captures only task constraints.
Supervision comes from hindsight labeling. For each real multi-turn conversation in LMSYS-USP (94k conversations) and WildChat, a strong labeler reads the full conversation. For each user turn it writes the note that best explains that turn given what the user eventually reveals later. Because the labeler knows the future, it can mark intent that was present but withheld. That is precisely the drip-feeding behavior (roughly 2.7% of intent keywords revealed per turn) that prompted simulators fail to produce. We filter the labels in two ways. First, an independent extractor must recover the conversation's implicit profile from the sequence of notes; this is the cycle-consistency check. Second, the note at turn $k$ must not mention facts the user states only after turn $k$ unless the note marks them as withheld.
Training has two stages. Supervised fine-tuning on (note, utterance) pairs comes first. It is followed by a short RL stage with a cycle-consistency reward, as in Know You First, computed on the generated utterances alone so the model cannot satisfy it through the note.
Baselines
The naive reference is a prompted instruction-tuned model given the same LMSYS-USP profile and a detailed persona instruction, generating turns directly. This is the standard simulator in UserRL-style environments. Beating it is necessary but uninformative.
The strongest existing competitor on realism is UserLM-8b from Flipping the Dialogue. That paper reports 60–70% lower perplexity on real human utterances than prompted assistants, per-turn intent-keyword disclosure of about 2.7%, and simulated users that are indistinguishable from humans roughly 80% of the time under the Pangram detector. On profile-conditioned realism, the strongest competitor is USP from Know You First and Be You Better. Human raters preferred it over profile-prompted GPT-4o for authenticity by 74% to 13%. To make the comparison fair, we re-train a UserLM-style direct simulator and a USP-style simulator from the same 8B base, on the same conversations, with the same number of training tokens and the same hyperparameter search budget as our model. Our model differs from these only in producing the note. Beating them means three things, all measured on held-out WildChat and on the out-of-domain PRISM data. We must match their realism in the first ten turns, within noise, on perplexity, disclosure rate, and AI-detectability. We must be clearly better on these metrics and on per-turn profile recoverability beyond turn twenty. And we must be less detectable as AI overall.
For drift control we include split-softmax from Measuring and Controlling Persona Drift and system-prompt repetition, each applied to the direct simulator with its scaling factor tuned on our validation split. On that paper's own benchmark, GPT-3.5-turbo-16k loses about 10% in instruction stability over 8–16 rounds, which calibrates how much drift to expect.
Three ablations isolate the key ingredient. The first is appended reasoning: the same notes are generated but all of them are kept in context. It tests whether overwriting matters. The second is a frozen note: the turn-one note is repeated and never updated. It tests whether updating matters, as opposed to having a persona reminder near the end of the input. The third is prompted notes without hindsight supervision, which tests whether the training signal matters.
Why not just…?
Why not just use UserLM-8b, which already sounds human? This is the real threat. UserLM-8b may already capture most of the realism that matters for short sessions. Our experiments are designed to show where it stops being enough. Our prediction is specific: its advantage over prompted models holds early, but its per-turn profile recoverability and trait expression fall as sessions lengthen, because nothing in the model carries accumulated state. If the direct model shows no decline out to 60 turns, the note adds cost without benefit. We would report that and drop the recurrence claim.
Why not just keep the reasoning appended, as normal chain of thought does? Appended reasoning is the most faithful version of the recurrence argument in Autoregressive + Chain of Thought = Recurrent. But it inflates the transcript, which is the cause of attention decay that the persona-drift paper identifies. The appended-note ablation settles this directly. Our hypothesis requires a crossover: appended reasoning should be at least as good early and worse late, with the crossover occurring earlier under smaller input-window budgets.
Why not just keep a model-written JSON memory, or a running summary? Best Friends, Not Forever found that such memory collapses to near-chance user-state recall. Our design differs in three respects, any of which could be the one that matters, and the evidence forces us to test them rather than assume them. The note is trained on hindsight labels rather than written zero-shot. It describes the speaker's own state rather than tracking a partner. And it is read before every utterance rather than retrieved occasionally. We run an untrained JSON-state baseline and a hierarchical-summary baseline under the same budget $B$. We also probe our note with multiple-choice questions about user state, modeled on that paper's trajectory probe. If our note shows the same near-chance recall, the approach has failed.
Why not just use a million-token context? The same companion study gave long-context models, including Gemini 1.5 Pro, the full transcript. Their average trajectory-memory accuracy was still only 44.4%. We nonetheless run the direct simulator with an unrestricted window as a reference point. The goal is to separate the effect of window size from the effect of state representation.
Proposed Experiments
We run realism evaluation on WildChat and PRISM using the metrics from Flipping the Dialogue. For long horizons, we extend conversations to 60 turns by pairing each simulator with fixed assistant models: GPT-4o-class models and an agent trained with UserRL. Every five turns we measure three things. The first is profile recoverability, meaning whether an independent extractor can recover the LMSYS-USP profile from a window of recent utterances. The second is assistant-likeness, meaning detector score and response length. The third is disclosure rate. We repeat this under input-window budgets of 4k, 8k, and 16k tokens.
Diversity is measured as how widely the simulated users' recovered profiles spread across the space of real profiles, reported separately for rare profile types. This matters because Know You First reports that assistants perform worse for underrepresented profiles.
For extrinsic validity we measure the drop in agent success on the math and coding tasks from Flipping the Dialogue, on IntentGym and TravelGym from UserRL, and on PersonaForge-Bench. A more realistic simulator should reproduce more of the human-versus-simulator gap of 74.6% to 57.4%. It should also rank agents in the same order that real users do. We check the ranking with a small human study of 200 sessions.
Two checks decide the outcome. The first is the recoverability-versus-turn curve for three conditions: direct, appended, and overwrite. The proposal succeeds only if overwrite stays flat while the other two decline, and if the crossover with appended reasoning moves earlier as the window budget shrinks. The second is a causal test of whether the note actually drives behavior. We splice in the note from a different user partway through a dialogue and measure whether the next utterances move toward the new user's profile. This responds to the finding in Critical or Compliant? that reasoning text can be rationalization rather than cause. If utterances ignore the spliced note, the gains come from training data rather than from state tracking, and we would say so.
References
- Flipping the Dialogue: Training and Evaluating User Language Models
- Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer
- Measuring and Controlling Persona Drift in Language Model Dialogs
- Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality
- UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
- CharacterBench: Benchmarking Character Customization of Large Language Models
- Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles
- Best Friends, Not Forever: Evaluating Long-Horizon Persona Collapse and Behavioral Drift in AI Companions
- BRIDGE: Triangular Fixed-Point Refinement for Long-Horizon Persona Consistency
- PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems
- SAGE : A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn Agent Evaluation
- Upfront Chain-of-Thought: A Cooperative Framework for Chain-of-Thought Compression
- MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines
- Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations