Chat models often get facts wrong late in a long conversation even though the correct fact was stated earlier. One reason is a shortcut: the model treats the most recent turn as the thing that settles the answer. In annotated dialogues the immediately preceding turn is a cause of the response about 90% of the time, yet 81% of responses also depend on something further back (Less is More). A model that has learned "the latest turn wins" will be right most of the time. It will also contradict itself whenever a recent aside, hypothetical, or quoted rumor conflicts with a fact established earlier.
This note proposes a test and a fix. Direct preference optimization (DPO) fine-tunes a model by raising the likelihood of a preferred reply relative to a rejected one, with no separate reward model (Direct Preference Optimization). The test asks whether ordinary DPO on natural dialogue data makes the recency shortcut worse. The fix trains DPO on constructed pairs in which recency carries no information about which reply is correct. In half the pairs the governing fact sits far back and a recent turn contains a non-authoritative conflicting claim. In the other half the recent turn is a genuine correction and does govern. The model can only prefer the right reply by reading what role each turn plays in the conversation, not where the turn sits.
Core Claim
We make two claims. The first is diagnostic. Standard DPO on naturally collected faithful-versus-hallucinated dialogue pairs steepens the relationship between evidence distance and hallucination rate, because in natural data recency correlates with correctness. This follows the argument of Causal DPO, which shows in recommendation that DPO's contrastive loss amplifies confounders present in the preference pairs. We transfer that argument to position in dialogue history.
The second claim is constructive. DPO on recency-balanced counterfactual pairs flattens that relationship. It also does not trade away the model's ability to accept real corrections. That trade-off is exactly what a naive "prefer distant evidence" dataset would produce.
The main measurement is a distance curve: context-conflicting hallucination rate plotted against the number of turns between the governing evidence and the question, reported separately for "earlier fact governs" and "later correction governs" cases. A flat curve at a low level for both cases is the target.
Relation to Prior Work
Three lines of work motivate the gap.
- DPO against hallucination in multimodal models. HDPO builds targeted negatives, including a "long context" type made by truncating captions. HA-DPO aligns vision-language models against object hallucination. Both are single-turn and image-grounded. Neither controls for where in a multi-turn history the grounding evidence sits.
- Snowballing in multi-turn dialogue. MM-Snowball documents error accumulation across turns: LLaVA-1.5-7B accuracy falls below 2% in turns 3–5 and recovers only when explicitly re-anchored. Its remedy is a training-free inference-time blend of visual features and logits. We change the policy through training instead, and we work in text.
- Faithful dialogue data. FaithDial cleans Wizard of Wikipedia, whose turns are over 60% hallucinated, and trains with a contrastive loss (InfoNCE) plus a separate discriminator. It never isolates how far back the knowledge lies.
Two works constrain the design.
- Pruning the history. Less is More shows that feeding the full history invites spurious dependence. Its fix is to prune non-causal turns with an external classifier that assumes at most two causes per response. We want the generator itself to learn which turns govern, without such a cap.
- Token-level DPO. TDPO shows that sequence-level DPO collapses output diversity: entropy is 3.196 for DPO versus 4.915 for TDPO2 on Anthropic HH. We therefore track diversity and include TDPO2 as a loss variant.
One work supplies a method we reuse. Causal DPO adds an invariance penalty across inferred environments. We test a lightweight analogue that uses distance buckets as environments. It is an ablation, not the core contribution.
We deliberately do not follow CDCR-SFT, which has the model write an explicit causal graph before answering. That approach targets formal causal-reasoning questions, not tracking facts across a conversation.
Research Program
Building the pairs
We build the preference data from three sources:
- FaithDial knowledge snippets;
- FaithEval passages (FaithEval);
- CGDIALOG-style conversations from ESConv and MSC, used as filler.
Each training dialogue contains a fact-establishing turn, a later conflicting turn, a variable number of filler turns, and a final question. The conflicting turn is one of two kinds:
- Non-authoritative. A hypothetical ("suppose it were…"), a quoted rumor, a joke, or another speaker's unverified guess. The earlier fact governs.
- Genuine correction or update from an authoritative speaker. The later turn governs.
A third kind of dialogue has no establishing fact at all. For these the preferred reply abstains, mirroring FaithEval's unanswerable condition.
Two quantities are sampled uniformly:
- The distance between the governing turn and the question, over 1, 2, 4, 8, 16, and 32 turns.
- Whether the governing turn is the earlier or the later one, set so that recency predicts correctness exactly 50% of the time.
The preferred reply follows the governing turn. The rejected reply is a fluent answer that follows the non-governing turn. Both replies are generated by the same model and differ only in the disputed span, so the pair isolates the factual choice from style.
Training
We fine-tune Llama-3.1-8B-Instruct with LoRA using the standard DPO loss on these pairs. We also run TDPO2 on the same pairs.
The invariance ablation adds a penalty on the variance, across distance buckets, of the mean implicit reward margin. The margin is $\beta$ times the difference in log-ratio between chosen and rejected replies, where $\beta$ is DPO's KL-strength coefficient. This asks the preference signal to be equally strong at every distance.
Analysis
To check whether the policy actually reads the governing turn, we measure attribution mass (gradient-times-input over history turns) on the governing turn versus the conflicting turn, bucketed by distance.
Baselines
Do nothing. The first reference point is the untouched instruction-tuned backbone, evaluated zero-shot on every distance curve. It shows how strong the recency shortcut is before any training.
Supervised fine-tuning. The second is supervised fine-tuning (SFT) on the preferred replies only, using the same pairs. It tests whether the negatives matter at all.
Standard DPO on natural pairs. This is the strongest natural competitor and the direct test of our diagnostic claim. We pair FaithDial's edited faithful turns as chosen against the original Wizard of Wikipedia hallucinated turns as rejected. This is the obvious way to apply DPO to dialogue hallucination, and it uses real rather than synthetic data.
Contrastive loss from FaithDial. We rerun FaithDial's InfoNCE objective on our backbone. The reported reference is 77.4% human-evaluated faithfulness for T5 with InfoNCE on FaithDial (FaithDial). Because that figure is for a different backbone, we compare against our own rerun, not against the number.
Token-level DPO. We run TDPO2 on the natural pairs. The reported figures are 67.33% alignment accuracy against 59.43% for DPO on Anthropic HH (TDPO). This baseline tests whether finer credit assignment alone fixes the distance curve without balanced data.
External pruning. The pipeline competitor is CONSTRAIN-style pruning in front of the base model (Less is More). No reported numeric MAUVE or hallucination result was found on Lacuna for CGDIALOG.
Ablations isolating the key ingredient. The first, and most important, is an unbalanced counterfactual set in which the earlier fact always governs. It uses identical construction, size, and compute, and differs only in the balance.
Fairness conditions. All trained methods share:
- the same backbone and LoRA rank;
- the same number of preference pairs (about 20k, matching HDPO's scale);
- the same number of optimizer steps;
- an equal budget of hyperparameter trials over $\beta$ and learning rate, selected on a shared held-out dev split.
What counts as winning. The proposal wins only if both of the following hold, while HaluEval dialogue discrimination and MT-Bench quality stay within noise of the SFT model:
- its distance-curve slope is significantly flatter than standard DPO's, in both the "earlier governs" and "later governs" conditions;
- its mean hallucination rate is lower than the best baseline's.
Why not just…?
Why not just prompt the model to re-read the whole conversation or reason step by step? MM-Snowball shows that explicit re-anchoring prompts do cause a rebound. FaithEval reports that chain-of-thought gives only marginal gains against strong parametric priors, with GPT-4o at 47.5% on counterfactual contexts. We include a re-anchoring prompt as an inference-time baseline on the distance curves. If it flattens them as much as training does, the training contribution shrinks to a cost argument. We would report that.
Why not just prune irrelevant turns or retrieve the relevant earlier turn? Pruning removes the aside, but it cannot by itself decide whether a later turn is an aside or a correction. That is the discourse judgment our balanced pairs target. Existing pruners also cap causes at two per response and depend on scarce causal-graph annotations. The CONSTRAIN baseline on the "later correction governs" split tests this directly. We expect pruning to help the aside condition and fail the correction condition.
Why not just run DPO with hard negatives, without the balancing? This is the real threat, since our method is hard-negative DPO. The unbalanced ablation settles it. If unbalanced pairs score as well on held-out genuine-correction dialogues, balancing is unnecessary. We predict instead that they teach "prefer distant evidence" and cause the model to reject real updates.
Why not just use SFT on the correct replies? SFT never shows the model the tempting wrong reply. It can therefore imitate correct answers without learning to suppress the recency shortcut. The SFT-on-chosen baseline, run with identical data, tests this.
Proposed Experiments
Main evaluation. The primary evaluation is held-out distance curves on constructed dialogues built from knowledge sources not seen in training. Three transfer checks support it:
- FaithEval passages embedded at varying turn depths, covering the counterfactual and unanswerable conditions;
- the HaluEval dialogue subset (HaluEval), where ChatGPT's detection accuracy is 72.40%;
- a text-only snowball protocol adapted from MM-Snowball. We inject an early model error, add a later user correction, and measure accuracy on turns 3–6.
Regression checks. MT-Bench and Anthropic HH verify that general quality holds. We also report reply entropy to catch the diversity collapse TDPO warns about.
Decisive checks. The diagnostic claim fails if standard DPO on natural FaithDial pairs does not steepen the distance slope relative to SFT. In that case the proposal becomes purely constructive. The constructive claim fails if either of the following happens:
- the balanced model loses to the unbalanced ablation on the aside condition;
- it loses to the prompting baseline across both conditions.
The attribution analysis should show attention mass shifting toward the governing turn regardless of its distance. If gains appear without that shift, we would suspect the model found a surface cue in our construction. We would then regenerate the conflicting turns with a different writer model and retest.
References
- Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery
- Reliable and Responsible Foundation Models
- Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
- Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
- MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
- FaithDial: A Faithful Benchmark for Information-Seeking Dialogue
- Token-level Direct Preference Optimization
- Mitigating Hallucinations in Large Language Models via Causal Reasoning
- FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models