Conversations With a Known Answer: Training and Stress-Testing Self-Correcting Dialogue Agents

Reinforcement learning (RL) that rewards only a correct final answer teaches language models to check and repair their own work. In math and code, models trained this way learn to stop, re-check, and restart without anyone showing them how. Conversational agents do not get this training signal. They are fine-tuned on transcripts that other LLMs wrote, then judged by LLM graders, and nothing in either step can tell whether the agent's view of the conversation is actually correct. We propose building dialogues whose correct outcome is known by construction. A program samples a hidden task graph of user goals, constraint changes, and contradictions. An LLM user simulator turns each scripted event into natural language. The correct final set of tool calls is then computable, which gives an RL reward for the agent and a fresh, uncontaminated test set every time the seed changes.

The same environment lets us ask a question that long-dialogue research has blurred: when an agent falls apart in a long conversation, is that because the conversation is long in tokens, or because it contains too many state changes to track? The first is the classic Transformer length-extrapolation failure. Past the training length, positional signals go out of distribution and attention spreads too thinly across the context. The second is a failure of the agent's policy. They call for different fixes. The procedural generator can vary token length and number of state updates independently, so we can separate them.

Core Claim

We claim three things. First, outcome-rewarded RL on procedurally generated, verifiable dialogues produces measurable self-correction in conversation: the agent revises earlier commitments when it notices they conflict with later information. Second, this learned behavior extends the number of state updates the agent can handle beyond those seen in training. Third, inference-time positional fixes help only along the token-length axis and not along the state-update axis.

If the third claim holds, much of what is reported as "long-context degradation" in dialogue is really a state-tracking failure. That would explain why passing retrieval tests does not predict multi-hop success. If it fails, and positional fixes close most of the gap, then the dialogue-horizon problem reduces to the already well-studied length-extrapolation problem. That outcome is also worth knowing.

Where This Sits

Three lines of work motivate the proposal. The self-correction work shows what outcome rewards can produce in single-answer settings:

  • DeepSeek-R1 shows verification behavior emerging from final-answer rewards.
  • Re² adds an explicit restart action, rewarded at the batch's estimated fresh-start success rate. It raises restarts from about 0.5% to over 30% and lifts Qwen2.5-7B-Instruct on AIME 2025 from 8.6% to 21.2%.
  • SimpleTIR extends outcome RL to multi-turn tool use. It discards "void turns" to stabilize training and reaches 50.5 on AIME24, up from 22.1.

All three need a mechanical answer checker. SimpleTIR caps interactions at 10 turns and names longer horizons as open. We supply a checker for conversation and push the horizon well past training.

Dialogue synthesis supplies the method:

  • ATOD samples goal co-occurrence graphs from Schema-Guided Dialogue and synthesizes dialogues with turn-level goal states (open, pending, completed, failed, abandoned). It uses them only to evaluate existing agents, through a memory-based evaluator.
  • Bao et al. parameterize simulated users with personality traits to add friction.
  • Bae et al. show that full-session generation beats single-turn pairs.

We reuse ATOD's graph sampling and persona-driven simulation. The key change is that the simulator becomes a renderer of scripted events rather than the source of truth, so the ground truth stays programmatic. That turns a static benchmark into a training environment.

The contamination work supplies the evaluation logic:

  • DyVal generates reasoning problems from directed acyclic graphs. Models with strong static scores, such as phi-1.5 and WizardMath-13B, score near zero on its dynamic instances.
  • A²RBench shows a 17.7% drop under symbolic remapping.

Both stay single-turn. We carry graph generation and remapping (renaming entities and slots) into multi-turn dialogue.

Length extrapolation work constrains the design and supplies the diagnostic probes:

  • SelfExtend maps distant positions into the trained range without retraining.
  • HeadScale ties failure past the training length to rising attention entropy and fixes it with per-head temperatures.
  • BiPE splits positions into within-segment and across-segment indices.
  • LongBench reports a 17% drop for GPT-3.5-Turbo-16k beyond 8k tokens.

We use these methods as treatments and probes, not as our contribution.

Research Program

The generator. A seed produces a goal graph over Schema-Guided Dialogue services, and a script of user events over that graph. Events include introducing a goal, changing a constraint ("make it Tuesday instead"), withdrawing a goal, contradicting an earlier statement, and referring back to a much earlier turn. An LLM user simulator with a sampled persona phrases each event in natural language.

A separate checker model confirms that each rendered utterance expresses its intended event, using the multi-sample agreement idea of DeepMath-103K. Failed renderings are regenerated. The correct final tool-call state is computed from the script alone.

Two control knobs are independent of each other:

  • State updates per dialogue ($K$): the number of events that change the correct final state.
  • Filler between updates: irrelevant chit-chat turns or long tool outputs. Filler sets total token length $T$ without changing $K$.

Entity and slot renaming at test time guards against the agent memorizing surface patterns.

Training. The agent is trained with group-relative policy optimization (GRPO), an RL method that scores each rollout against others sampled for the same prompt. The reward is the fraction of goals whose final tool-call state matches the oracle, with a penalty for spurious calls. Training uses $K \le 8$ and $T$ within the base model's pretrained window.

We give the agent the option to issue an explicit "revise earlier booking" action. We then measure how often it uses the action, rather than requiring it.

Self-correction measurement. We measure self-correction two ways. The natural measure is the rate at which a wrong intermediate tool state is later repaired. The controlled probe injects a mistaken agent turn into the history at a chosen point, a prefilled error the agent never made. We then record whether and when the agent repairs it. The probe separates "can recover from an error" from "rarely errs."

Mechanism. Across the $K \times T$ grid, we log per-head attention entropy (following HeadScale) at the turns where the agent must use an early fact. This tells us whether failures beyond the window coincide with attention spreading out.

Baselines

Naive reference. The untrained instruct model, prompted with the tool schema. It shows how much of the task is solved without training and how sharply it degrades along each axis.

Strongest training competitor. Supervised fine-tuning on oracle-repaired transcripts, matched to the RL run. The generator knows the correct state, so we can produce teacher transcripts where a strong LLM makes a mistake and fixes it, with the fixes checked against the oracle. This is the conversational version of LIMO, which reports 63.3% on AIME24 from 800 curated chains.

Fairness rules:

  • Both methods see the same seeds and the same $K$ and $T$ ranges.
  • Both get the same total generated tokens during training.
  • Both get the same hyperparameter search budget.

RL beats this baseline only if it is better on final-state exact match at $K$ values beyond training, and on the injected-error repair rate, by a margin larger than the spread across three seeds.

Strongest structured competitor. An ATOD-style explicit state tracker. The agent keeps a structured goal-status memory that it rewrites each turn, and its context is truncated to recent turns plus that memory. ATOD reports its memory evaluator at 86.49% goal-detection F1 and 84.28% status-tracking accuracy on complex dialogues. That is the best reported state-tracking figure in this file, though it measures tracking, not agent success.

Positional treatments. SelfExtend and HeadScale-style temperature tuning, applied to both the RL agent and the supervised agent. The best figure on file is SelfExtend's 100% passkey retrieval at 24k tokens on Mistral-7B.

Missing reference. No reported result was found on Lacuna for any agent trained with outcome RL on verifiable multi-turn dialogue. The untrained and supervised baselines above serve as that reference.

Key-ingredient ablation. RL with the identical reward but with all correction events (constraint changes, contradictions) removed from training scripts. If self-repair still emerges, the reward alone drives it. If it disappears, exposure to the need for revision is the ingredient.

Why not just…?

Why not just fine-tune on teacher dialogues that contain corrections? This is the most serious threat. LIMO shows that small curated supervised sets can match or beat large-scale training. On the other side, Chen et al. and Parallel-R1 report that supervised imitation copies the surface form of reasoning and breaks out of distribution. Our matched baseline settles it, measured at unseen $K$ and with injected errors placed where teacher transcripts never placed them. If supervision matches RL there, we will report that the environment's value lies in verification and evaluation rather than in RL.

Why not just apply SelfExtend or another positional fix to long conversations? SelfExtend acknowledges that grouping blurs fine position as contexts grow. LongBench shows multi-hop reasoning degrading even where retrieval succeeds. The $K \times T$ grid tests this directly. We predict positional fixes recover accuracy along $T$ at fixed $K$ but not along $K$ at fixed $T$. If they recover both, our second claim is false.

Why not just give the agent an external state memory? It may well make horizon length irrelevant, and ATOD's tracker numbers are strong. However, LongBench finds that retrieval and chunking can hurt models with good native context, because they drop interdependent signals spread across turns. Contradictions that refer back to early turns are exactly such signals. We include the memory agent as a full competitor. We also test RL combined with memory, since the useful finding may be that RL teaches an agent to keep its own memory accurate.

Why not just evaluate on ATOD as released? A fixed, published test set becomes training data over time. It also cannot vary $K$ and $T$ independently. Our generator draws fresh seeds and remaps entities. Following DyVal, we check freshness by comparing scores on released versus new seeds.

Decisive Checks

The central experiment is the full $K \times T$ grid on two base models:

  • Llama-2-7B-chat, which has a 4k window and so lets $T$ cross the pretrained limit.
  • Qwen2.5-7B-Instruct, whose longer window makes it the within-window control.

Each cell evaluates every system on final-state exact match and injected-error repair rate.

The proposal succeeds if three things hold. RL beats matched supervised training at $K$ beyond 8 on held-out, remapped seeds. Positional treatments close the $T$-axis gap but leave the $K$-axis gap. The rise in attention entropy appears only in cells past the window.

A secondary check tests whether the "longer reasoning, lower accuracy" pattern that Re² reports appears in dialogue. If it does, we ask whether it lines up with crossing the window or with error position. That tells us whether overthinking in conversation has a positional cause at all.