Abstract: Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, Songlin Hu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
In retrieval-based dialogue systems, the challenge lies in selecting the most appropriate response from a candidate pool based on the conversation history. While context is essential for understanding user intent, it can be a "double-edged sword." Long conversation histories often introduce noise—irrelevant turns that can mislead models. Traditional architectures, such as SMN and DAM, typically match candidate responses against every previous turn, making them fragile when a candidate response contains distractor words that overlap with irrelevant parts of the history.
The authors demonstrate that adding just one or two random words to a candidate response can cause popular models to make errors that a human would easily avoid. To mitigate this, they propose the Multi-hop Selector Network (MSN), which explicitly filters out irrelevant conversation history before the matching process begins.
Methodology
The MSN architecture implements a "select-then-match" pipeline to improve robustness and accuracy. The model consists of four key modules:
- Attentive Module: Uses a variant of Multi-head Attention to create dense word-level representations for each utterance and the candidate response.
- Multi-hop Selector: This module acts as a filter, using the most recent utterance as a "key" to determine the relevance of all previous turns. It employs both word-level and utterance-level selectors. The multi-hop mechanism allows the model to iteratively find relevant context, even when the final utterance is uninformative (e.g., "Okay").
- Context Fusion: This component aggregates relevance scores across multiple hops and applies a threshold ($\gamma$). Utterances with scores below this threshold are effectively discarded or down-weighted.
- Matching & Aggregation: The filtered context is matched against the candidate response using Origin, Self, and Cross matching strategies. The results are then aggregated via a CNN and GRU to produce a final ranking score.
Experiments and Results
MSN was evaluated on three large-scale benchmarks: the Ubuntu Corpus (technical support), the Douban Corpus (social dialogue), and an E-commerce Corpus (customer service).
- State-of-the-Art Results: MSN outperformed all baselines across all datasets. On the Ubuntu Corpus, it achieved an R10@1 of 80.0%. On the E-commerce dataset, it reached 60.6%, a significant jump from the 50.1% achieved by the DUA model.
- Ablation Success: Removing the selector module caused the R10@1 score on the E-commerce dataset to drop from 60.6% to 55.4%, proving the filtering step's importance.
- Adversarial Robustness: MSN demonstrated significantly higher resilience to adversarial noise compared to previous SOTA models, which saw performance drops of 7-13% when distractors were introduced.
Conclusion
The Multi-hop Selector Network addresses the problem of noise in multi-turn dialogues by making context management an explicit, interpretable step. By filtering irrelevant utterances before matching, MSN achieves superior performance and robustness. Future research may explore methods to ensure logical consistency between selected context and responses, as well as automated tuning for hyperparameters like the number of hops and the selection threshold.
Summary updated