TL;DR
Researchers have introduced MSDialog, a new dataset of over 2,000 multi-turn technical support conversations annotated with fine-grained user intents. By analyzing these interactions, the study maps out the "DNA" of information-seeking—revealing how users and agents navigate from initial questions to successful resolutions through predictable patterns of feedback and clarification.
What Problem Does This Paper Address?
While conversational assistants (CAs) like Siri and Alexa excel at simple, one-off tasks (e.g., "What’s the weather?"), they struggle with complex, multi-turn information seeking. Real-world troubleshooting or research involves a back-and-forth: the agent asks for details, the user provides feedback, and the solution is refined over several steps.
The major bottleneck in building smarter CAs has been a lack of high-quality, labeled data. Existing datasets are either too informal (like IRC chats), not focused on information seeking (like Twitter), or lack labels for why a person is saying what they are saying (their "intent"). This paper fills that gap by providing a structured look at how humans help each other solve problems.
Methods
The authors developed a multi-stage pipeline to create and analyze the MSDialog dataset:
- Data Collection: They crawled 35,000 threads from the Microsoft Community forum, filtering for high-quality interactions (3–10 turns, 2–4 participants, and a confirmed "correct answer").
- Taxonomy Design: They defined a 12-class taxonomy for user intent, including categories like Clarifying Question, Potential Answer, Further Details, and Negative Feedback.
- Crowdsourced Annotation: Using Amazon Mechanical Turk, they had workers label 10,000 individual utterances. Crucially, they allowed multiple labels per utterance, recognizing that a single sentence can serve multiple purposes (e.g., a greeting combined with a follow-up question).
- Flow Analysis: The researchers used Markov Models to visualize the "flow" of conversations, identifying the most common transitions between intents (e.g., how often a Potential Answer leads to Positive Feedback).
Datasets
- MSDialog (New): 2,199 multi-turn QA dialogs (10,020 utterances) from Microsoft technical forums. It is specifically designed for intent analysis and response ranking.
- Ubuntu Dialog Corpus (UDC): Used as a comparison point. While much larger, UDC is unlabeled and consists of informal IRC-style chat, which contains more noise and slang compared to the moderated MSDialog.
Key Insights and Ideas
- The "Politeness" Factor: Greetings/Gratitude and Potential Answer were the most frequent labels. This highlights that human-to-human technical support is deeply social; maintaining rapport is as central to the conversation as the technical solution itself.
- Negative Feedback is Constructive: One of the most interesting findings was the high co-occurrence of Negative Feedback and Further Details. When a user says a solution didn't work, they almost always explain why or provide more context. This "failure" turn is actually a critical pivot point for the agent to provide a better answer.
-
The Standard Path: The researchers identified a "typical" conversation path:
Original Question → Potential Answer → Further Details → Potential Answer → Positive Feedback → Terminal. -
Cross-Platform Consistency: Despite the informal nature of the Ubuntu Dialog Corpus (UDC), the researchers found that the underlying intent patterns were remarkably similar to MSDialog. This suggests that the "logic" of seeking information is somewhat universal, regardless of the platform or language style.
Key Results
- Dataset Scale: 2,199 dialogs with an average of 4.56 turns per dialog.
- Label Density: Each utterance has an average of 1.83 labels, proving that human speech is multi-functional.
- Intent Distribution: Greetings/Gratitude (22%) and Potential Answer (22%) dominate the interactions, while Repeat Question is the rarest (<10%).
- Flow Patterns: The Markov Model analysis successfully visualized how Information Requests and Clarifying Questions act as bridges to Further Details, which eventually lead to successful resolutions.
Limitations and Future Work
The authors acknowledge that MSDialog is currently limited to the technical support domain on a moderated forum. Real-world spoken conversations or interactions in other domains (like medical or legal) might exhibit different intent dynamics.
The next step for this research is to use the MSDialog dataset to train neural architectures for intent prediction. If a conversational agent can accurately predict what a user is trying to achieve in the next turn, it can proactively provide better answers or ask more relevant clarifying questions.
Summary updated