Introduction
In multi-turn dialogue systems, distinguishing between relevant history and conversational "noise"—such as typos, filler words, or off-topic remarks—is a significant challenge. While humans naturally filter these distractions to focus on the core topic, conversational AI models often struggle, treating every utterance with similar weight. This lack of selective focus often leads to degraded response quality.
This paper addresses the problem by introducing the Distracting Attention Score (DAS) ratio, a metric designed to evaluate context attention distribution, and a training strategy utilizing self-contained distractions to teach models to ignore irrelevant input. Experimental results demonstrate a 10% performance improvement on the proposed metric.
Methodology
The authors propose a two-pronged approach involving a new evaluation metric and an optimization strategy.
The DAS Ratio
The Distracting Attention Score (DAS) ratio measures a model's ability to allocate focus. During evaluation, irrelevant "distracting utterances" are inserted into the dialogue history. The DAS ratio is calculated as the average attention score given to these distractions divided by the attention given to the actual conversation history. A lower DAS ratio indicates a more robust model that successfully ignores noise.
Optimization: Self-Contained Distractions
To improve attention distribution, the researchers implemented a multi-task learning strategy during training. They randomly inserted distractions (utterances from other dialogues) into the training context and applied a specific attention loss term. This loss forces the model to minimize the attention weight assigned to any utterance marked as a distraction.
The strategy was tested across several LSTM-based architectures, including hierarchical models with static and dynamic attention mechanisms. As shown in the architecture diagram, the attention loss is applied to different components depending on the model type (e.g., non-hierarchical vs. hierarchical with Utterance Integration LSTMs).
Experimental Setup
The study utilized the Ubuntu chatlogs dataset, which contains approximately 0.48 million dialogues. This dataset is ideal for this research as it combines technical troubleshooting with casual conversation. To rigorously test the models, the researchers created three types of distracting test sets based on the nature of the inserted noise:
- Random: Utterances sampled randomly from the corpus.
- Frequent: Utterances composed of common words (e.g., "Why should I help you?").
- Rare: Utterances containing infrequent vocabulary (e.g., "Would you have lunch?").
As illustrated above, these distractions were placed at the beginning, middle, or end of the dialogue history to account for structural biases in the models.
Results and Discussion
Key Insights
- Perplexity is Deceptive: The study found that models with nearly identical perplexity scores can have vastly different abilities to manage context. The DAS ratio reveals these hidden weaknesses.
- The "Middle" Matters: Due to the natural recency bias of LSTMs, testing distractions in the middle of a conversation proved most effective for measuring true attention distribution.
- Optimal Noise Level: Training with a 0.7 probability of distraction insertion yielded the best results. A 1.0 probability caused models to erroneously assume noise is a mandatory part of every dialogue.
Performance Gains
The proposed optimization strategy improved the DAS ratio by approximately 10% across various model types. The "StaticUI" model—a hierarchical architecture with an Utterance Integration LSTM—combined with distraction training achieved the highest overall performance and demonstrated robustness against all noise types (Random, Frequent, and Rare).
Conclusion
This research highlights the importance of measuring and optimizing context attention in multi-turn dialogue systems. By using self-contained distractions and the DAS ratio, models can be trained to filter out conversational noise more effectively. While this study focused on LSTM architectures, the methodology provides a foundation for future work on Transformer-based models and more complex, adversarial distractions.
Summary updated