A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots

Yu Wu, Wei Wu, Chen Xing, Can Xu, Zhoujun Li, Ming Zhou

Abstract

We study the problem of response selection for multi-turn conversation in retrieval-based chatbots. The task requires matching a response candidate with a conversation context, whose challenges include how to recognize important parts of the context, and how to model the relationships among utterances in the context. Existing matching methods may lose important information in contexts as we can interpret them with a unified framework in which contexts are transformed to fixed-length vectors without any interaction with responses before matching. The analysis motivates us to propose a new matching framework that can sufficiently carry the important information in contexts to matching and model the relationships among utterances at the same time. The new framework, which we call a sequential matching framework (SMF), lets each utterance in a context interacts with a response candidate at the first step and transforms the pair to a matching vector. The matching vectors are then accumulated following the order of the utterances in the context with a recurrent neural network (RNN) which models the relationships among the utterances. The context-response matching is finally calculated with the hidden states of the RNN. Under SMF, we propose a sequential convolutional network and sequential attention network and conduct experiments on two public data sets to test their performance. Experimental results show that both models can significantly outperform the state-of-the-art matching methods. We also show that the models are interpretable with visualizations that provide us insights on how they capture and leverage the important information in contexts for matching.

Introduction

Retrieval-based chatbots aim to select the most appropriate response from a pre-defined index given a conversation history. In multi-turn scenarios, this task is particularly challenging because the "context" is not just the last message, but a sequence of previous utterances. Identifying which parts of the history are relevant to a candidate response requires modeling both fine-grained keyword matches and the chronological relationships between turns.

An example of a multi-turn conversation context where key information (e.g., "drum") is spread across multiple utterances.

As illustrated in the example above, a successful model must recognize that "drum" is a key topic introduced early in the conversation to correctly evaluate response candidates for the final turn.

Limitations of Existing Models

The authors identify two major flaws in existing architectures, which they categorize under a unified "representation-based" framework. As shown in Figure 1, these models typically transform the entire context into a fixed-length vector before any interaction with the candidate response occurs.

The unified framework of existing models, where context is compressed into a single vector before matching with a response.

This approach suffers from two main drawbacks:

  1. Information Loss: Compressing multiple utterances into a single vector often "washes out" critical keywords or phrases.
  2. Weak Interaction: Because the context and response only "meet" at the final matching step, the model cannot use the response as a "lens" to help identify which specific parts of the history are actually relevant.

The Sequential Matching Framework (SMF)

To address these issues, the authors propose the Sequential Matching Framework (SMF). Unlike previous models, SMF allows each individual utterance to interact with the response candidate at the earliest stage. The framework consists of three layers:

  1. Utterance-Response Matching: Each utterance in the context is matched with the candidate response immediately. The authors propose two specific architectures:
  2. Sequential Convolutional Network (SCN): Constructs word-word and segment-segment similarity matrices, then uses a CNN to extract features.
  3. Sequential Attention Network (SAN): Uses an attention mechanism where the response "attends" to important parts of the utterance to build a weighted representation.
  4. Matching Accumulation: A Recurrent Neural Network (specifically a GRU) takes the sequence of matching vectors from the first layer and accumulates them. This allows the model to "remember" the order of turns and filter out irrelevant ones using the GRU's gating mechanism.
  5. Prediction: The final hidden state of the GRU is used to calculate the matching score.

Experimental Evaluation

The models were evaluated on two major public benchmarks: the Ubuntu Dialogue Corpus (technical support logs) and the Douban Conversation Corpus (an open-domain Chinese forum featuring a human-labeled test set).

The SMF-based models (SCN and SAN) significantly outperformed all baselines. On the Ubuntu corpus, SAN achieved an $R_{10}@1$ of 0.733, a roughly 7% improvement over the best baseline. On the Douban corpus, SAN achieved a Mean Average Precision (MAP) of 0.536, outperforming the Multi-View model (0.505). Visualizations confirmed that the models successfully "highlight" important keywords (like "unzip" or "files") while ignoring functional stop-words.

Conclusion and Future Work

The SMF paradigm demonstrates that early interaction between context and response is crucial for multi-turn dialogue. By treating the context as a sequence of interactions rather than a single monolithic representation, models can better capture the nuances of human conversation.

Despite these gains, challenges remain. The models still struggle with logical consistency—for instance, failing to recognize when a response contradicts a previously established premise. Future work will involve exploring more sophisticated negative sampling during training and improving the quality of the initial candidate retrieval step.

Summary updated

Abstract

We study the problem of response selection for multi-turn conversation in retrieval-based chatbots. The task requires matching a response candidate with a conversation context, whose challenges include how to recognize important parts of the context, and how to model the relationships among utterances in the context. Existing matching methods may lose important information in contexts as we can interpret them with a unified framework in which contexts are transformed to fixed-length vectors without any interaction with responses before matching. The analysis motivates us to propose a new matching framework that can sufficiently carry the important information in contexts to matching and model the relationships among utterances at the same time. The new framework, which we call a sequential matching framework (SMF), lets each utterance in a context interacts with a response candidate at the first step and transforms the pair to a matching vector. The matching vectors are then accumulated following the order of the utterances in the context with a recurrent neural network (RNN) which models the relationships among the utterances. The context-response matching is finally calculated with the hidden states of the RNN. Under SMF, we propose a sequential convolutional network and sequential attention network and conduct experiments on two public data sets to test their performance. Experimental results show that both models can significantly outperform the state-of-the-art matching methods. We also show that the models are interpretable with visualizations that provide us insights on how they capture and leverage the important information in contexts for matching.
BibTeX
@article{wu2019sequential,
  author = {Yu Wu and Wei Wu and Chen Xing and Can Xu and Zhoujun Li and Ming Zhou},
  title = {{A Sequential Matching Framework for Multi-turn Response Selection in Retrieval-based Chatbots}},
  journal = {Comput. Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00345},
}
Figure