A Sequential Matching Framework for Multi-Turn Response Selection in Retrieval-Based Chatbots

Yu Wu, Wei Wu, Chen Xing, Can Xu, Zhoujun Li, Ming Zhou

Introduction: The Challenge of Multi-Turn Conversations

Retrieval-based chatbots aim to select the most appropriate response from a pre-indexed database. While single-turn response selection—matching a single message to a single response—is a relatively mature field, multi-turn response selection presents significant hurdles. In a multi-turn scenario, the chatbot must consider the entire conversation history (the context) to make an informed choice.

As illustrated in Table 1, identifying the correct response requires the model to extract relevant information across multiple turns while ignoring noise. For instance, generic greetings like "How are you doing?" are less important than specific topics like "drum class" or "free first lesson."

An example of a multi-turn conversation where context relevance is key to selecting the correct response candidate.

The authors identify two primary challenges:

  1. Information Extraction: Determining which parts of a long history are relevant to the current response candidate (e.g., focusing on "Python help" while ignoring "Hello").
  2. Utterance Relationships: Modeling the dependencies and chronological order between different turns to capture the flow of dialogue.

Moving Beyond the "Represent-then-Match" Paradigm

Existing models typically follow what the authors call a "Unified Framework." In this approach, the entire context is compressed into a single fixed-length vector before it is compared to the response candidate. This "represent-then-match" strategy often results in the loss of fine-grained information. Because the context is aggregated in a vacuum, the response candidate cannot "guide" the model to focus on the most salient parts of the history.

The traditional 'represent-then-match' framework used by existing models, where context is compressed before matching.

The Sequential Matching Framework (SMF)

To address these limitations, the authors propose the Sequential Matching Framework (SMF). This paradigm shifts from "represent-then-match" to a "match-then-accumulate" approach. Instead of compressing the context first, SMF matches the response candidate with each individual utterance in the context at the very first step.

The framework consists of three distinct layers:

  1. Utterance-Response Matching: Each utterance $u_i$ in the context interacts with the response candidate $r$ to form a matching vector. The authors propose two specific architectures for this layer:
  2. Sequential Convolutional Network (SCN): This model builds 2D similarity matrices (capturing word-to-word and segment-to-segment similarities) and uses a Convolutional Neural Network (CNN) to extract features.
  3. Sequential Attention Network (SAN): This model employs an attention mechanism where the response "attends" to important segments of the utterance, creating a weighted representation that highlights relevant information.
  4. Matching Accumulation: A Recurrent Neural Network (specifically a Gated Recurrent Unit, or GRU) processes these matching vectors in chronological order. This allows the model to "remember" the evolution of the conversation.
  5. Matching Prediction: The final hidden state of the GRU is used to calculate the ultimate matching score between the context and the candidate.

By matching at the utterance level early on, the model uses the response candidate as a filter, identifying relevant keywords or phrases in the history that might otherwise be lost during global compression.

Experimental Results and Insights

The SMF-based models were evaluated on two large-scale benchmarks: the Ubuntu Dialogue Corpus (English technical support) and the Douban Conversation Corpus (Chinese open-domain).

The results demonstrated a significant leap in performance:

  • Ubuntu Corpus: The SAN model achieved an $R_{10}@1$ of 0.733, outperforming the best baseline at the time by 7 percentage points.
  • Douban Corpus: SAN achieved a Mean Average Precision (MAP) of 0.536, a 3.1 percentage point improvement over previous state-of-the-art models like Multi-View and DL2R.

Beyond raw metrics, the authors found that the models offered better interpretability. Visualizing the GRU's internal "gates" revealed that the framework successfully learned to down-weight noisy or generic utterances while focusing on information-dense turns that defined the topic.

Conclusion and Future Directions

The Sequential Matching Framework represents a significant advancement in retrieval-based dialogue systems by prioritizing early interaction between context and candidates. However, challenges remain. Current models still struggle with logical consistency—they may prioritize semantic similarity even if a response contradicts a previous statement. Additionally, there is a performance gap when moving from random negative samples in training to "hard" negatives retrieved by search engines in real-world testing. Future work will focus on bridging this "retrieval gap" and incorporating logical reasoning into the matching process.

Summary updated

Abstract

We study the problem of response selection for multi-turn conversation in retrieval-based chatbots. The task involves matching a response candidate with a conversation context, the challenges for which include how to recognize important parts of the context, and how to model the relationships among utterances in the context. Existing matching methods may lose important information in contexts as we can interpret them with a unified framework in which contexts are transformed to fixed-length vectors without any interaction with responses before matching. This motivates us to propose a new matching framework that can sufficiently carry important information in contexts to matching and model relationships among utterances at the same time. The new framework, which we call a sequential matching framework (SMF), lets each utterance in a context interact with a response candidate at the first step and transforms the pair to a matching vector. The matching vectors are then accumulated following the order of the utterances in the context with a recurrent neural network (RNN) that models relationships among utterances. Context-response matching is then calculated with the hidden states of the RNN. Under SMF, we propose a sequential convolutional network and sequential attention network and conduct experiments on two public data sets to test their performance. Experiment results show that both models can significantly outperform state-of-the-art matching methods. We also show that the models are interpretable with visualizations that provide us insights on how they capture and leverage important information in contexts for matching.
BibTeX
@article{wu2019sequential,
  author = {Yu Wu and Wei Wu and Chen Xing and Can Xu and Zhoujun Li and Ming Zhou},
  title = {{A Sequential Matching Framework for Multi-Turn Response Selection in Retrieval-Based Chatbots}},
  journal = {Comput. Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00345},
}
Figure