Abstract
Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.
Introduction
While conversational assistants (CAs) like Siri and Alexa excel at simple, one-off tasks, they often struggle with complex, multi-turn information seeking. Real-world troubleshooting or research involves a dynamic back-and-forth: the agent asks for details, the user provides feedback, and the solution is refined over several steps. The major bottleneck in building smarter CAs has been a lack of high-quality, labeled data. Existing datasets are often too informal or lack labels for user intent. This paper fills that gap by providing a structured look at how humans help each other solve problems through the MSDialog dataset.
The MSDialog Dataset
The researchers developed a multi-stage pipeline to create and analyze the MSDialog dataset, crawling 35,000 threads from the Microsoft Community forum. They filtered for high-quality interactions (3–10 turns, 2–4 participants, and a confirmed "correct answer"). A 12-class taxonomy for user intent was defined, including categories like Clarifying Question, Potential Answer, Further Details, and Negative Feedback.
Using Amazon Mechanical Turk, 10,000 individual utterances were labeled. Crucially, multiple labels per utterance were allowed, recognizing that a single sentence can serve multiple purposes, such as a greeting combined with a follow-up question. The resulting dataset consists of 2,199 multi-turn QA dialogs (10,020 utterances) specifically designed for intent analysis and response ranking.
Intent Distribution and Key Insights
Analysis of the dataset reveals that human-to-human technical support is deeply social. Greetings/Gratitude (GG) and Potential Answer (PA) were the most frequent labels, each accounting for 22% of the distribution. This highlights that maintaining rapport is as central to the conversation as the technical solution itself.
One significant finding was the high co-occurrence of Negative Feedback and Further Details. When a user indicates a solution did not work, they almost always provide more context or explain why. This "failure" turn serves as a critical pivot point for the agent to provide a better answer.
Conversation Flow Patterns
To visualize the "DNA" of information-seeking, the researchers used Markov Models to identify common transitions between intents. They identified a "standard path" for successful resolutions: Original Question → Potential Answer → Further Details → Potential Answer → Positive Feedback → Terminal.
The flow analysis successfully visualized how Information Requests and Clarifying Questions act as bridges to Further Details, which eventually lead to successful resolutions. Despite the informal nature of other corpora like the Ubuntu Dialog Corpus, underlying intent patterns were remarkably similar, suggesting a universal logic to seeking information.
Conclusion and Future Work
MSDialog provides a foundation for training neural architectures for intent prediction. If a conversational agent can accurately predict what a user is trying to achieve in the next turn, it can proactively provide better answers or ask more relevant clarifying questions. Future work will expand this analysis beyond technical support to other domains like medical or legal interactions.
Summary updated