Analyzing and Characterizing User Intent in Information-seeking Conversations

Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, Minghui Qiu

Abstract

Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.

Introduction

While conversational assistants (CAs) like Siri and Alexa excel at simple, one-off tasks, they often struggle with complex, multi-turn information seeking. Real-world troubleshooting or research involves a dynamic back-and-forth: the agent asks for details, the user provides feedback, and the solution is refined over several steps. The major bottleneck in building smarter CAs has been a lack of high-quality, labeled data. Existing datasets are often too informal or lack labels for user intent. This paper fills that gap by providing a structured look at how humans help each other solve problems through the MSDialog dataset.

The MSDialog Dataset

The researchers developed a multi-stage pipeline to create and analyze the MSDialog dataset, crawling 35,000 threads from the Microsoft Community forum. They filtered for high-quality interactions (3–10 turns, 2–4 participants, and a confirmed "correct answer"). A 12-class taxonomy for user intent was defined, including categories like Clarifying Question, Potential Answer, Further Details, and Negative Feedback.

Using Amazon Mechanical Turk, 10,000 individual utterances were labeled. Crucially, multiple labels per utterance were allowed, recognizing that a single sentence can serve multiple purposes, such as a greeting combined with a follow-up question. The resulting dataset consists of 2,199 multi-turn QA dialogs (10,020 utterances) specifically designed for intent analysis and response ranking.

Intent Distribution and Key Insights

Analysis of the dataset reveals that human-to-human technical support is deeply social. Greetings/Gratitude (GG) and Potential Answer (PA) were the most frequent labels, each accounting for 22% of the distribution. This highlights that maintaining rapport is as central to the conversation as the technical solution itself.

Figure 1: Distribution of user intents in the MSDialog dataset.

One significant finding was the high co-occurrence of Negative Feedback and Further Details. When a user indicates a solution did not work, they almost always provide more context or explain why. This "failure" turn serves as a critical pivot point for the agent to provide a better answer.

Conversation Flow Patterns

To visualize the "DNA" of information-seeking, the researchers used Markov Models to identify common transitions between intents. They identified a "standard path" for successful resolutions: Original Question → Potential Answer → Further Details → Potential Answer → Positive Feedback → Terminal.

Figure 4: Markov Model visualizing the flow of user intents during a conversation.

The flow analysis successfully visualized how Information Requests and Clarifying Questions act as bridges to Further Details, which eventually lead to successful resolutions. Despite the informal nature of other corpora like the Ubuntu Dialog Corpus, underlying intent patterns were remarkably similar, suggesting a universal logic to seeking information.

Conclusion and Future Work

MSDialog provides a foundation for training neural architectures for intent prediction. If a conversational agent can accurately predict what a user is trying to achieve in the next turn, it can proactively provide better answers or ask more relevant clarifying questions. Future work will expand this analysis beyond technical support to other domains like medical or legal interactions.

Summary updated

Abstract

Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.
BibTeX
@inproceedings{qu2018analyzing,
  author = {Chen Qu and Liu Yang and W. Bruce Croft and Johanne R. Trippas and Yongfeng Zhang and Minghui Qiu},
  title = {{Analyzing and Characterizing User Intent in Information-seeking Conversations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2018},
  doi = {10.1145/3209978.3210124},
}
Figure