Managing Contextual Noise in Multi-Turn Dialogue History

291 concepts192 papers3 proposals

In human conversation, we rarely speak in isolated, self-contained sentences. We use pronouns, omit implied words, and shift topics fluidly. To understand the current turn of a conversation, an AI system must look backward at the dialogue history.

The most straightforward way to grant a model this historical sight is naive concatenation: gluing all previous utterances together into a single text block and feeding it into an encoder. While this approach preserves all past information, it introduces a critical vulnerability: contextual noise accumulation. As a conversation progresses, the raw text block fills with conversational fillers, social pleasantries, and outdated topics. This accumulation dilutes the dense signal of the user's immediate intent, distracting attention mechanisms and degrading downstream performance.


The Mechanism of Attention Dilution

Why does naive concatenation fail when transformer models possess such massive context windows? The issue is not capacity, but selectivity.

When a dialogue history grows long and unstructured, the information density drops. The core entities and intents required to answer the current query become buried under layers of peripheral chatter. In retrieval-based conversational assistants, for instance, Yucheng Li et al. observe that including raw histories introduces excessive noise from non-essential dialogue, which misleads retrieval engines and lowers precision (LINBER: A Retrieval-based Conversational Assistant using Entity Linking and BERT).

Naive Concatenation:
[Turn 1: Hello!] -> [Turn 2: I need a hotel.] -> [Turn 3: Actually, let's look for a restaurant first.] -> [Current Turn: Is there an Italian place nearby?]
                                                                                                       ^
                                                                        Model must filter out "hotel" to focus on "Italian restaurant"

This problem is particularly acute in task-oriented systems like Dialogue State Tracking (DST) or Text-to-SQL parsing. In these settings, the model must map natural language to structured slots or database schemas. As shown by Jiaying Hu et al., long histories obscure specific task-relevant values, making it difficult for models to maintain focus on the exact parameters required for state tracking (SAS: Dialogue State Tracking via Slot Attention and Slot Information Sharing).

When a model calculates attention weights over a massive, unrefined sequence, the softmax function distributes probability mass across hundreds of tokens. If a significant portion of those tokens consists of irrelevant prose or outdated schema links from previous turns, the representation of the current turn drifts (Ziqiang Liu et al., TP-Link: Fine-grained Pre-Training for Text-to-SQL Parsing with Linking Information). The model essentially "forgets" what is currently important because it is forced to look at everything that came before.


Explicit Selection vs. Implicit Weighting

A common counterargument is that deep neural networks should naturally learn to ignore noise through their trained weights. However, Chunyuan Yuan et al. demonstrate that managing context purely through implicit neural weights makes it difficult to interpret or control which parts of a history influence a decision (Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots). Without explicit filtering, dominant but outdated topics easily skew response selection.

To combat this, researchers have shifted toward architectures that actively select or isolate useful historical turns before processing.

Impact-Driven Turn Selection

Instead of assuming all past turns are relevant, models can evaluate the actual utility of each turn. Fengran Mo et al. developed a pseudo-labeling approach that selects historical turns based on their actual retrieval impact (Learning to Relate to Previous Turns in Conversational Search). By filtering out turns that do not actively contribute to clarifying the current intent, the system prevents noise from unrelated queries from confusing the search engine during topic shifts.

Intent-Driven Masking

Another approach isolates the user's core objective to shield the model from historical noise. Zihao Yi et al. show that masking historical context and focusing strictly on the current turn's intent results in cleaner semantic representations (Intent-driven In-context Learning for Few-shot Dialogue State Tracking). This filtering significantly boosts the quality of example retrieval in few-shot learning setups, proving that a smaller, denoised context window can outperform a larger, noisy one.


Structural Distillation and Memory Decay

If we cannot simply throw away the past, how do we preserve it without the accompanying noise? The solution lies in structural distillation—transforming raw conversational text into more compact, abstract representations.

Raw Dialogue History  --->  Abstractive Summary  --->  Entity/Proposition List
(High Noise, Low Density)                               (Low Noise, High Density)

Malsha V. Perera et al. formalize this via an adaptive context management framework that dynamically transitions older conversation turns into summaries, and eventually into simple entity lists (Towards Adaptive Context Management for Intelligent Conversational Question Answering). This ensures that the model retains the underlying concepts required for conversational coherence without drowning in raw prose.

Empirical evidence supports this hybrid approach. Vishakh Padmakumar et al. found that while flat concatenation is a resilient baseline for short sessions, a hybrid strategy that pairs an abstractive summary of the distant past with the raw text of the most recent utterances consistently wins out in long-form conversations (Investigating the Representation of Open Domain Dialogue Context for Transformer Models).

For structured tasks like Text-to-SQL, the distillation can be modeled as a graph where older nodes fade away. Binyuan Hui et al. introduced a "memory decay" mechanism within an evolving graph structure (Dynamic Hybrid Relation Exploration Network for Cross-Domain Context-Dependent Semantic Parsing). By actively forcing the model to forget distant historical details, they prevent outdated context from distracting the model from the current user intent.


Fine-Grained Denoising via Soft Masking

When rewriting incomplete utterances (resolving ellipsis and co-references), models often pull words from irrelevant parts of the history. Operating purely at the sentence level can be too coarse, while operating purely at the token level introduces too much noise.

To bridge this gap, Haowei Du et al. proposed a multi-granularity framework (Multi-Granularity Information Interaction Framework for Incomplete Utterance Rewriting). The system first calculates relevance scores at the sentence level to identify which past turns are actually related to the current turn. It then uses these macro-scores to softly mask a token-level edit matrix.

$$\text{Masked Attention} = \text{Attention}(Q, K) \odot \sigma(\text{Sentence Relevance})$$

By gating token-level interactions with sentence-level relevance, the model is physically prevented from attending to noisy entities in distant, unrelated turns. This dual-layer filtering preserves precision and prevents the generation of hallucinated or off-topic text.

Ultimately, as conversational agents tackle longer horizons and more complex workflows, managing context density becomes just as critical as expanding model capacity. Filtering and reconstructing dialogue history to match the specific requirements of the current task phase ensures that the model's reasoning engine remains focused on the signal, not the noise (Bin Lei et al., InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction).

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 194 papers
  2. Version 1 · 2026-10-04 · 250 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dialogue_history_noise_42263062,
  title = {dialogue history noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dialogue-history-noise/rdr_42263062d87942cc8c90c3d8f8f9c91f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_dialogue_history_noise_42263062_v2,
  title = {dialogue history noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dialogue-history-noise/rdr_42263062d87942cc8c90c3d8f8f9c91f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake