Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation

Liu Yang, Hamed Zamani, Yongfeng Zhang, Jiafeng Guo, W. Bruce CroftarXiv 2017 · OpenReview

Abstract

The recent boom of AI has seen the emergence of many human-computer conversation systems such as Google Assistant, Microsoft Cortana, Amazon Echo and Apple Siri. We introduce and formalize the task of predicting questions in conversations, where the goal is to predict the new question that the user will ask, given the past conversational context. This task can be modeled as a "sequence matching" problem, where two sequences are given and the aim is to learn a model that maps any pair of sequences to a matching probability. Neural matching models, which adopt deep neural networks to learn sequence representations and matching scores, have attracted immense research interests of information retrieval and natural language processing communities. In this paper, we first study neural matching models for the question retrieval task that has been widely explored in the literature, whereas the effectiveness of neural models for this task is relatively unstudied. We further evaluate the neural matching models in the next question prediction task in conversations. We have used the publicly available Quora data and Ubuntu chat logs in our experiments. Our evaluations investigate the potential of neural matching models with representation learning for question retrieval and next question prediction in conversations. Experimental results show that neural matching models perform well for both tasks.

Introduction

Most current conversational agents are reactive, responding only after a user command. To advance toward proactive AI, systems must anticipate user needs. This research addresses two critical tasks: Question Retrieval, finding semantically similar questions in large archives to leverage human-answered content, and Next Question Prediction, anticipating the user's next query based on chat history to enable pre-fetching or better intent understanding.

A primary challenge in these tasks is the "lexical gap"—the phenomenon where users express the same intent using different vocabulary. Traditional Information Retrieval (IR) models like BM25 often fail here because they rely on exact keyword overlaps.

Proposed Architecture: LSTM-CNN-Match

The authors propose a hybrid neural architecture designed to map pairs of sequences to a matching probability.

The LSTM-CNN-Match architecture for sequence matching.

The model consists of several key components:

  1. LSTM Layer: Unlike models using raw embeddings, this architecture first processes text through a Long Short-Term Memory (LSTM) layer to capture long-term dependencies and contextual word meanings.
  2. Convolutional Layer (CNN): The LSTM output is processed by a CNN with multiple filter sizes (1 to 5), capturing n-gram features at various scales.
  3. Max Pooling: This layer extracts the most salient features from the convolutional maps.
  4. Combined Model: To handle specific matches (like dates or numbers) that neural networks might miss, the authors used LambdaMART to integrate the neural score with traditional features like BM25 and word overlap counts.

Experimental Setup

The model was evaluated on two datasets:

  • Quora Question Pairs: Over 400,000 question pairs for the retrieval task.
  • Ubuntu IRC Chat Logs: Processed to extract conversational contexts and subsequent user questions for the prediction task.

Key Insights: Bridging the Lexical Gap

Neural models excel by operating in a continuous vector space, allowing them to recognize semantic similarities that keyword-based models miss. For instance, as shown in the retrieval examples, the neural model correctly identifies the relationship between "movies" and "shows" or "good" and "best."

Comparison of retrieval results showing the neural model's ability to bridge the lexical gap.

The study found that:

  • Context is Crucial: The LSTM layer significantly improved performance over CNN-only models, indicating that word order and context are vital for understanding question intent.
  • Task Difficulty: Predicting the next question in a conversation is substantially harder than retrieving similar questions. Conversational contexts are longer and the relationship between history and future questions is more complex than simple paraphrasing.
  • Hybrid Superiority: The "Combined Model" achieved the best results, proving that deep learning and traditional IR features are complementary.

Results and Conclusion

In the Question Retrieval task (Quora), the LSTM-CNN-Match model achieved an MRR of 0.880, outperforming BM25 (0.861). The Combined Model reached a state-of-the-art MRR of 0.894. For Next Question Prediction (Ubuntu), neural models again outperformed traditional baselines, though simpler CNN-Match variants performed better on recall metrics for long contexts.

Future work includes exploring attention mechanisms to better handle long conversation histories and moving toward generative models that can synthesize new questions rather than just retrieving them from a fixed set.

Summary updated

Abstract

The recent boom of AI has seen the emergence of many human-computer conversation systems such as Google Assistant, Microsoft Cortana, Amazon Echo and Apple Siri. We introduce and formalize the task of predicting questions in conversations, where the goal is to predict the new question that the user will ask, given the past conversational context. This task can be modeled as a "sequence matching" problem, where two sequences are given and the aim is to learn a model that maps any pair of sequences to a matching probability. Neural matching models, which adopt deep neural networks to learn sequence representations and matching scores, have attracted immense research interests of information retrieval and natural language processing communities. In this paper, we first study neural matching models for the question retrieval task that has been widely explored in the literature, whereas the effectiveness of neural models for this task is relatively unstudied. We further evaluate the neural matching models in the next question prediction task in conversations. We have used the publicly available Quora data and Ubuntu chat logs in our experiments. Our evaluations investigate the potential of neural matching models with representation learning for question retrieval and next question prediction in conversations. Experimental results show that neural matching models perform well for both tasks.
BibTeX
@misc{yang2017neural,
  author = {Liu Yang and Hamed Zamani and Yongfeng Zhang and Jiafeng Guo and W. Bruce Croft},
  title = {{Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation}},
  year = {2017},
  eprint = {1707.05409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.05409},
}
Figure