Training Language Models to Ignore Distracting Turns Without Forgetting Early Ones

Language models are trained and tested mostly one exchange at a time. They get a prompt, they answer, and the answer is scored. Real use is a conversation, and conversations fill up with material that is irrelevant to the current request: side topics, abandoned sub-tasks, and the model's own earlier answers. Two findings from separate literatures meet here. Semantically coherent but irrelevant sentences can cut accuracy by more than 45% even on frontier models, and the model's own generated text distracts it most. Separately, agents copy their own past turns instead of reacting to new information. The one training fix known to work against distraction, preference optimization that contrasts clean and distracted prompts, has only ever been tested on single-turn questions.

This note proposes to test whether that single-turn robustness carries over to conversations, and expects it will not. It then proposes a training signal built for dialogue. For each turn, the preferred response is the one the model itself produces when the history keeps every relevant turn, including the very first, and removes only the distractor turns. The dispreferred response is the one it produces under the full cluttered history. The model should learn to discount irrelevant turns while still honoring constraints set many turns earlier. Existing fixes give up one of these two properties. Clipping the history loses the early constraints, and summarizing it invents claims.

Core Claim

Robustness to distraction learned from single-turn contrasts does not survive distractors spread across many dialogue turns. A preference signal that uses relevance-filtered history as the target does survive. It flattens the accuracy-versus-distractor-turns curve and does not lose turn-1 constraints. Here "relevance-filtered history" means the dialogue with only the known-irrelevant turns deleted, which is possible because we construct the dialogues and know which turns are distractors. This differs from the closest work in what counts as the "good" context. Mitigating Conversational Inertia in Multi-Turn Agents trains with Context Preference Learning (CPL). CPL is DPO (direct preference optimization, a method that trains a model to prefer one of two responses) whose chosen response comes from recent-only context, so it teaches the model to down-weight distant turns in general, including the ones that carry the user's goal. Breaking Focus contrasts clean and perturbed single prompts and never sees accumulation across turns. Our target keeps distance and removes only irrelevance.

Research Program

The work has three parts.

The first part is a diagnostic benchmark, which we call Multi-Turn Distraction. It turns questions from MMLU, CommonsenseQA, GSM8K and MATH-500 into dialogues. Turn 1 states a durable constraint, such as an answer format, a unit, or an assumption that changes the answer. The premises of the question are split across later turns. Between them we insert $k \in \{0, 2, 4, 8, 16\}$ distractor turns. Distractors come in three kinds. The first is static off-topic chat. The second is adaptive distractors from the tree-search procedure in Adaptive Distraction, which filters candidates with a classifier and uses a proxy model to steer perturbations toward errors; we adapt it to write whole turns rather than single sentences. The third is self-generated distractors, meaning turns written by the target model itself. That paper's "self-challenge paradox" predicts the third kind will be the most damaging, and in dialogue this overlaps with the self-imitation that Mitigating Conversational Inertia traces to "diagonal attention."

A necessary control makes some mid-dialogue turns genuine updates, where the user revises a constraint. A model that simply learns "ignore the middle of the conversation" will then fail visibly. This control guards against the over-pruning failure that the research file flags for context compressors.

The second part is the transfer study. We train Llama-3.1-8B and Qwen3-8B with single-turn clean-versus-distracted DPO, reproducing the recipe in Breaking Focus. We then measure accuracy as a function of $k$. The motivating concern comes from Fine-Tuning Large Language Models with Sequential Instructions and A Closer Look at the Limitations of Instruction Tuning, which show that single-turn tuning generalizes poorly to multi-step structure. The attention-dilution effect in Multi-Turn Jailbreaking via Attention Shifting suggests a mechanism: benign turns drain attention away from the critical tokens.

The third part is relevance-filtered history preference training. For each training dialogue and turn $t$, we sample the model's response $y^+$ given the history $h_t^{\text{rel}}$, which is the full history minus the distractor turns and keeps turn 1 and any genuine updates. We sample $y^-$ given the full history $h_t$. We keep only pairs where a task checker scores $y^+$ as correct and $y^-$ as wrong. We then train with the standard DPO objective, with both responses conditioned on the full history:

$$\mathcal{L} = -\log \sigma\!\left(\beta\left[\log\frac{\pi_\theta(y^+\mid h_t)}{\pi_{\text{ref}}(y^+\mid h_t)} - \log\frac{\pi_\theta(y^-\mid h_t)}{\pi_{\text{ref}}(y^-\mid h_t)}\right]\right)$$

Here $\pi_\theta$ is the model being trained, $\pi_{\text{ref}}$ is the frozen starting model, $\sigma$ is the logistic function, and $\beta$ sets how far the model may move from the reference. The model thus learns to behave on cluttered history as it would on clean history. At inference it needs no filter, no extra model call, and no change to the cache of stored attention keys and values, which preserves the key-value cache reuse that Mitigating Conversational Inertia shows matters for speed.

As a mechanism check, we measure the attention mass that the answer tokens place on turn-1 tokens, on distractor tokens, and on the model's own previous turns, before and after training. This is the analysis that the file notes ACR and Drift No More? did not perform.

Baselines

The naive reference is the base instruction-tuned model on full raw history, with and without the system prompt "ignore irrelevant information." That prompt matters because Adaptive Distraction reports it gives almost no protection. The anchor number is that paper's report that adaptive distractors drop GPT-4o-mini on CommonsenseQA from 85.7% to 22.0%, and that the average drop across models exceeds 45%.

The first training baseline is single-turn distraction DPO as in Breaking Focus. That paper reports roughly 50% improvement on perturbed inputs for smaller models. The file gives no absolute accuracy for it, so we will reproduce it.

The strongest existing method for the dialogue setting is CPL, optionally combined with Clip Context, from Mitigating Conversational Inertia. It reports that CPL plus Clip Context raises Qwen3-8B's average success across eight AgentGym tasks from 64.9% to 72.5%. Among published methods it is the one closest to our training signal.

The inference-time baselines are periodic goal reminders, following Drift No More?, which reports up to an 11.8% reduction in divergence on τ-bench retail for LLaMA-3.1-70B, and LLM summarization of history as in CORAL. On CORAL, summarization raises citation precision from 18.1% to 31.1% for Qwen2.5-7B-SFT.

All trained variants start from the same checkpoint. They use the same underlying questions, the same number of preference pairs, the same LoRA rank, the same $\beta$ sweep, and the same number of gradient steps. Only the construction of the chosen and rejected responses differs.

Two ablations isolate the key ingredient. In the first, the chosen response comes from recent-only context, which is CPL's target applied to our data. In the second, it comes from the single clean prompt with no history. We will count our method as beating CPL if two things hold together. On Multi-Turn Distraction at $k=16$, it must be at least as accurate as CPL. On turn-1 constraint compliance, it must be clearly better. Winning on accuracy alone does not count. The oracle reference, which is the base model given $h_t^{\text{rel}}$ at test time, is the ceiling we hope to approach.

Why not just…?

Why not just apply single-turn distraction DPO, which already works? It may transfer, and the transfer study tests this first. If the single-turn model's curve stays flat as $k$ grows and it keeps turn-1 constraints, the multi-turn training signal is unnecessary. We will then report the positive transfer result, which by itself fills the untested cell the file identifies. Our expectation that it fails rests on the sequential-tuning evidence and the attention-dilution result cited above, but only this experiment can settle it.

Why not just use CPL with Clip Context? It is fast and already proven on agents. However, its chosen responses come from recent context by construction. That rewards discounting distance itself, and the authors themselves note that clipping loses early instructions. Our turn-1 constraint probes and genuine-update controls measure exactly this cost. If CPL shows no loss of turn-1 compliance, our advantage shrinks to robustness against self-generated distractors, and we will say so.

Why not just filter or summarize the history at inference, as ACR or Self-MAP do? ACR uses a learned router with a noise-filtering operator. On the Multi-turn Instruction Following for Conversational Web Agents finds that stripping noise is its single most important component. This is a real threat, because an accurate filter would approach the oracle. The costs are an extra model call per turn, broken cache reuse whenever the prefix changes, and the over-claiming by summaries that Mitigating Conversational Inertia reports. We will run a trained relevance classifier at matched parameter budget as a baseline. If it matches our accuracy at similar latency, the honest conclusion is that filtering suffices.

Why not just add goal reminders? They are nearly free. But they only lower the level at which drift settles, in a setting where the noise is random rather than targeted, and they cannot stop self-imitation. We include them and also stack them on our method.

Proposed Experiments

The decisive check is the accuracy-versus-$k$ curve on Multi-Turn Distraction for all variants, reported separately for static, adaptive, and self-generated distractors. Alongside it we report turn-1 constraint compliance and compliance with genuine mid-dialogue updates. The self-generated condition tests whether the self-challenge paradox extends to dialogue. Held-out distractor-generation seeds and held-out source datasets check that the model has not just memorized what distractors look like.

For transfer beyond the synthetic setting, we evaluate without further tuning on the AgentGym suite against CPL's reported 64.9% → 72.5%, on τ-bench retail and airline, on CORAL citation precision against the 31.1% summarization number, and on MSC-QA, where C-DIC reports 10.4% against 4.2% for full prompting. Long-range recall on MSC-QA is where a method that down-weights distance would show harm.

As a safety-side check, we measure attack success rate on the multi-turn QuestionList attacks from ASJA, which reports 81.33% against undefended models. This asks whether learning to discount benign padding also removes the attention dilution that those attacks exploit.

The attention analysis should show that answer tokens place more attention on turn-1 and update turns, and less on distractor turns and on the model's own earlier turns. A method that improves accuracy without this shift would call the proposed mechanism into question.