Analyzing and Characterizing User Intent in Information-seeking Conversations

Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, Minghui Qiu

TL;DR

Researchers have introduced MSDialog, a new dataset of over 2,000 multi-turn technical support conversations annotated with fine-grained user intents. By analyzing these interactions, the study maps out the "DNA" of information-seeking—revealing how users and agents navigate from initial questions to successful resolutions through predictable patterns of feedback and clarification.

What Problem Does This Paper Address?

While conversational assistants (CAs) like Siri and Alexa excel at simple, one-off tasks (e.g., "What’s the weather?"), they struggle with complex, multi-turn information seeking. Real-world troubleshooting or research involves a back-and-forth: the agent asks for details, the user provides feedback, and the solution is refined over several steps.

The major bottleneck in building smarter CAs has been a lack of high-quality, labeled data. Existing datasets are either too informal (like IRC chats), not focused on information seeking (like Twitter), or lack labels for why a person is saying what they are saying (their "intent"). This paper fills that gap by providing a structured look at how humans help each other solve problems.

Methods

The authors developed a multi-stage pipeline to create and analyze the MSDialog dataset:

  1. Data Collection: They crawled 35,000 threads from the Microsoft Community forum, filtering for high-quality interactions (3–10 turns, 2–4 participants, and a confirmed "correct answer").
  2. Taxonomy Design: They defined a 12-class taxonomy for user intent, including categories like Clarifying Question, Potential Answer, Further Details, and Negative Feedback.
  3. Crowdsourced Annotation: Using Amazon Mechanical Turk, they had workers label 10,000 individual utterances. Crucially, they allowed multiple labels per utterance, recognizing that a single sentence can serve multiple purposes (e.g., a greeting combined with a follow-up question).
  4. Flow Analysis: The researchers used Markov Models to visualize the "flow" of conversations, identifying the most common transitions between intents (e.g., how often a Potential Answer leads to Positive Feedback).

Datasets

  • MSDialog (New): 2,199 multi-turn QA dialogs (10,020 utterances) from Microsoft technical forums. It is specifically designed for intent analysis and response ranking.
  • Ubuntu Dialog Corpus (UDC): Used as a comparison point. While much larger, UDC is unlabeled and consists of informal IRC-style chat, which contains more noise and slang compared to the moderated MSDialog.

Key Insights and Ideas

  • The "Politeness" Factor: Greetings/Gratitude and Potential Answer were the most frequent labels. This highlights that human-to-human technical support is deeply social; maintaining rapport is as central to the conversation as the technical solution itself.
  • Negative Feedback is Constructive: One of the most interesting findings was the high co-occurrence of Negative Feedback and Further Details. When a user says a solution didn't work, they almost always explain why or provide more context. This "failure" turn is actually a critical pivot point for the agent to provide a better answer.
  • The Standard Path: The researchers identified a "typical" conversation path: Original Question → Potential Answer → Further Details → Potential Answer → Positive Feedback → Terminal.

  • Cross-Platform Consistency: Despite the informal nature of the Ubuntu Dialog Corpus (UDC), the researchers found that the underlying intent patterns were remarkably similar to MSDialog. This suggests that the "logic" of seeking information is somewhat universal, regardless of the platform or language style.

Key Results

  • Dataset Scale: 2,199 dialogs with an average of 4.56 turns per dialog.
  • Label Density: Each utterance has an average of 1.83 labels, proving that human speech is multi-functional.
  • Intent Distribution: Greetings/Gratitude (22%) and Potential Answer (22%) dominate the interactions, while Repeat Question is the rarest (<10%).
  • Flow Patterns: The Markov Model analysis successfully visualized how Information Requests and Clarifying Questions act as bridges to Further Details, which eventually lead to successful resolutions.

Limitations and Future Work

The authors acknowledge that MSDialog is currently limited to the technical support domain on a moderated forum. Real-world spoken conversations or interactions in other domains (like medical or legal) might exhibit different intent dynamics.

The next step for this research is to use the MSDialog dataset to train neural architectures for intent prediction. If a conversational agent can accurately predict what a user is trying to achieve in the next turn, it can proactively provide better answers or ask more relevant clarifying questions.

Summary updated

Abstract

Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.
BibTeX
@inproceedings{qu2018analyzing,
  author = {Chen Qu and Liu Yang and W. Bruce Croft and Johanne R. Trippas and Yongfeng Zhang and Minghui Qiu},
  title = {{Analyzing and Characterizing User Intent in Information-seeking Conversations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2018},
  doi = {10.1145/3209978.3210124},
}

Loading related content...

Figure