When a language model answers a question about a conversation, it often reports the right fact but assigns it to the wrong person. It says Alice promised the refund when Bob did, or it treats a user's guess as the expert's confirmation. The root is simple. In dialogue, words like "I" and "you" change referent every turn, and a model reasoning in free text paraphrases each speaker's claim before using it. Every paraphrase is a chance to swap the speaker. We propose a decoding rule for the model's intermediate reasoning. Whenever a reasoning step cites what a participant said, that citation must be produced as a verbatim, speaker-labelled pointer into the transcript. The model generates it under a prefix-tree constraint, meaning it can only emit token sequences that exist as real utterances of that speaker. Everything else, including commonsense inference, the model writes freely. Under this rule, misattributed evidence cannot occur by construction, and no verifier has to be called on each step.
The second claim matters as much as the first. Once evidence must be pinned to real utterances, the reasoning chain has a natural stopping point: the question's entities are covered by cited utterances that connect through the conversation's reply structure. We expect this to shorten reasoning and to reduce the long-chain drift that makes extended thinking hurt on commonsense and situational tasks. The rule uses only structure that every transcript has (speaker labels, turn order, and reply links where available). The same decoder should therefore transfer without retraining from Ubuntu chat logs to TV scripts to exam dialogues.
Where this sits in prior work
Three lines of work motivate the idea and one supplies the method.
On the motivation side, speaker confusion is well documented. DREAM built a dialogue reading-comprehension exam and found that accuracy falls as dialogues get longer. Its best system reached 59.5% against 95.5% for humans. Is ChatGPT a Good Multi-Party Conversation Solver? showed that LLMs are "contextually blind" to flat multi-party transcripts. Adding [Speaker]: and Reply to #[ID] markers helped GPT-4 but hurt gpt-3.5-turbo, which the authors describe as an information-overload effect. The hallucination literature names the failure directly: entity and role swapping, for example conflating patient and donor records (Léo Milecki). Our proposal treats these results as evidence that speaker structure matters, and that pouring more metadata into the prompt is not a reliable fix.
The method comes from Graph-constrained Reasoning, known as GCR. GCR compiles knowledge-graph paths into a prefix tree and lets a fine-tuned LLaMA-3-8B sample only valid paths. It reports a 100% faithful-path ratio, against 48–62% without the constraint. GCR has two known weaknesses. It depends on an external entity linker, and it fails when the knowledge graph is missing edges. Dialogue removes both problems. The "graph" is the transcript itself, speaker labels come with the data, and all attributable evidence is present by definition. We do not constrain final answers or knowledge-base facts, only speaker-attributed premises. GCR constrains the whole reasoning path over a static factual graph, while we constrain only the evidence citations and leave inference steps free.
Two lines constrain the design. Hallucination Detection from Structural Reasoning Model, known as FlowScore, parses reasoning into a graph of conditions, steps and sources and verifies each step locally. It reports 76.59% average precision on MATH and 83.08% on HumanEval, but at the cost of one judge call per node. It also falsely flags steps that rely on unstated commonsense. We take the lesson that local, source-linked checking works. We move the check into decoding, where it costs a prefix-tree lookup instead of an LLM call. We also deliberately leave commonsense steps unconstrained, because roughly a third of DREAM questions need outside knowledge.
The literature on reasoning length is the second constraint. It shows that more thinking is not free. Don't Overthink it found that the shortest sampled chain can be up to 34.5% more accurate than the longest. TurtleBench found o1-preview at 84.40% below GPT-4o at 87.92% on situational puzzles. Are Reasoning Models More Prone to Hallucination? documents two failures: the model repeats a flawed premise, and its answer contradicts its own reasoning. Our stopping rule is a structural answer to these failures, not a length penalty.
Transfer is a framing constraint borrowed from Towards Foundation Models for Knowledge Graph Reasoning (ULTRA). ULTRA transfers across 57 knowledge graphs, with 0.395 average mean reciprocal rank zero-shot versus 0.344 for supervised models. It does so by reasoning only over how relations connect, never over their names. We apply the same principle by conditioning only on speaker identity and reply topology, never on dataset-specific labels. Supervised dialogue graph models, GLGR and DARER, are trained per dataset and rely on external role-labelling, coreference and discourse parsers. Molweni reports that automatic discourse parsing reaches only 54.8% F1 on joint link and relation prediction. That figure is why we avoid any predicted discourse relations.
Core Claim
Most speaker-misattribution errors in dialogue reasoning enter through free-text paraphrase of evidence. If every speaker-attributed premise is instead a constrained, verbatim, speaker-tagged pointer, two things should follow on multi-party dialogue QA without task-specific training:
- The rate of wrong-speaker reasoning should fall to near zero.
- Answer accuracy should rise, and reasoning should get shorter, compared with prompting the same model with speaker metadata and unconstrained chain-of-thought.
The accuracy gain should concentrate on questions whose answer changes when speakers are swapped.
Research Program
For each transcript we build a per-speaker prefix tree over utterances. An entry has the form ⟨cite A, u12⟩ "…verbatim span…". Spans may be any contiguous sub-span of a turn, so the model can quote the relevant clause without pasting whole turns. First-person words inside a quoted span keep their surface form. The pointer carries the resolved speaker, so a later free-text step can refer to "A" unambiguously.
The model reasons in ordinary text and opens a citation with a special token. From that token until the citation closes, decoding is masked to valid continuations of the prefix tree. The model therefore first commits to a speaker and turn, then to a span that speaker actually uttered. This commitment happens inside a single reasoning stream, with no parallel sampling.
The stopping rule works like this. We extract the question's content entities by string match against the transcript. We then allow the model to emit its answer only once two conditions hold. First, the cited utterances cover those entities. Second, the cited utterances are connected through reply links, or through adjacency when a dataset has no reply annotations. Before that point, the answer token is masked. After it, a small budget of further tokens is allowed. This resembles the early-termination relation in DiffKG, which stops a walk over a knowledge graph once an answer entity is reached. Here the walk runs over the conversation.
We run everything zero-shot with open instruction-tuned models, since constrained decoding requires access to the model's token probabilities. A light variant fine-tunes the model on its own successful constrained traces, following the recipe of training on short successful traces from Don't Overthink it. That variant trains on Molweni only and is tested elsewhere, which measures transfer.
The analysis centres on a speaker-swap diagnostic. For each test question we create a counterfactual transcript by exchanging the labels of the two speakers most relevant to the answer, and we recompute the gold answer where it changes. A model that tracks speakers should change its answer exactly when the gold answer changes. We also label reasoning chains for misattribution: a premise credited to a speaker who did not say it. Labelling is by a string check for constrained citations and by human annotation on a sample of free-text chains.
Baselines
The naive reference point is the same open model given the flat transcript and asked to answer directly. It sets the floor and tells us how much speaker confusion there is to fix.
The strongest prompting competitor is the structured speaker-and-addressee prompt from Is ChatGPT a Good Multi-Party Conversation Solver?, combined with free chain-of-thought. It is the strongest because it gives the model exactly the information our constraint uses, only as text. That paper reports GPT-4's response-selection accuracy rising from 53.73% to 57.46% with speaker information, and GPT-4 reaching 82.50% on addressee recognition on Ubuntu IRC. We run it with the same model, the same few-shot examples and the same maximum token budget as our method, tuning prompt wording on the same development split. Beating it means higher answer F1 or accuracy together with a lower misattribution rate at equal or fewer reasoning tokens. Beating it on the speaker-swap subset alone would still be informative.
For efficiency-aware reasoning, the strongest competitor is short-m@k from Don't Overthink it. It runs k parallel chains and takes a majority vote over the first m to finish, and it reports matching or beating standard majority voting with 40% fewer thinking tokens. We give it the same total token budget as our single stream, so it gets more samples, not more tokens per sample.
For verification, we run a FlowScore-style pipeline (Hallucination Detection from Structural Reasoning Model). The model emits a conditions/steps/sources graph, a judge checks each step against its cited turns, and failing chains are resampled. We report the judge calls this pipeline consumes.
As supervised references, not zero-shot competitors, we cite GLGR's reported 73.6% F1 on Molweni and 77.2% F1 on FriendsQA. We also cite BERT-wwm at 67.7% F1 and humans at 80.2% F1 on Molweni, all from the file. We found no reported zero-shot LLM result on Molweni or FriendsQA on Lacuna, so our own flat-prompt runs establish that number.
Two ablations isolate the key ingredient. The first uses the identical prompt, citation format and stopping rule but no decoding mask, so the model writes citations freely. This separates the effect of the citation format from the effect of the constraint. The second keeps the mask but removes the stopping rule, which isolates the claim about reasoning length.
Why not just…?
Why not just give a strong model speaker and addressee tags and let it reason? This is the most serious threat. For GPT-4-class models it may close most of the gap, since the prompting paper shows GPT-4 using structural tags well. The same paper shows weaker models degrading under that metadata, which suggests tags help only above a capability threshold. Our experiments run the comparison at three model sizes. If structured prompting matches our method on misattribution rate and speaker-swap consistency at the largest size, we will report the constraint as a remedy for small and mid-size models rather than a general one.
Why not just extract quotes first and answer second? A two-stage pipeline, where the model lists supporting utterances and then answers from them, captures some of the benefit. Its first stage still generates speaker labels and paraphrases freely, so it can still misattribute. It also cannot add evidence midway once reasoning reveals a need for another turn. We run it as a baseline with a constrained first stage, which isolates interleaving: if it matches the full method, then interleaving adds nothing and the contribution reduces to constrained extraction. We would say so.
Why not just detect misattributions after the fact and regenerate? Post-hoc checkers such as FactAlign align triplets and reach 0.889 F1 on WikiBio, and FlowScore checks each step. However, FactAlign extracts only named-entity triplets and was tested on monologue text. FlowScore pays one judge call per step and misfires on unstated commonsense. The FlowScore-style baseline measures whether detect-and-retry reaches our accuracy, and at what cost in judge calls.
Why not just shorten reasoning with short-m@k? Shortening addresses drift but not attribution. A short chain can still swap speakers. Comparing short-m@k and our method on the speaker-swap subset separates the two effects.
Proposed Experiments and Decisive Checks
We evaluate on the test sets of Molweni, FriendsQA and DREAM. Molweni's unanswerable questions let us measure fabrication directly, as the rate of answering when the gold answer is "no answer". Answers on the span-extraction datasets are scored by F1 after mapping generated answers to transcript spans. DREAM is scored by multiple-choice accuracy, against its reported 59.5% best ensemble. We report four quantities:
- Answer quality.
- Misattribution rate per chain.
- Speaker-swap consistency.
- Reasoning tokens per question.
For transfer, we fine-tune only on Molweni and test on FriendsQA and DREAM.
Three results would decide the proposal:
- Constraint check. Against the unmasked-citation ablation, the misattribution rate must drop substantially while answer quality does not fall. Otherwise the constraint merely forbids useful paraphrase.
- Gain location. Gains over structured prompting must be larger on the speaker-swap subset than on questions insensitive to speakers. If they are uniform, the mechanism is not doing what we claim.
- Stopping rule. Reasoning must get shorter without losing accuracy on DREAM's commonsense questions. The constraint deliberately does not cover those questions, so this checks whether pinning evidence curbs drift even where it cannot verify the inference.
If the first check passes and the second does not, the finding is still useful: attribution errors would then not be the main source of error in dialogue QA.
References
- DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension
- Is ChatGPT a Good Multi-Party Conversation Solver?
- MEDIMP: 3D Medical Images and clinical Prompts for renal transplant representation learning
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
- Hallucination Detection from Structural Reasoning Model
- Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning
- TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles
- Are Reasoning Models More Prone to Hallucination?
- Towards Foundation Models for Knowledge Graph Reasoning
- GLGR: Question-aware Global-to-Local Graph Reasoning for Multi-party Dialogue Reading Comprehension
- Relational Temporal Graph Reasoning for Dual-Task Dialogue Language Understanding
- Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure
- Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems
- FactAlign: Fact-Level Hallucination Detection and Classification Through Knowledge Graph Alignment