Towards a Model for Spoken Conversational Search

Johanne R. Trippas, Damiano Spina, Paul Thomas, Mark Sanderson, Hideo Joho, Lawrence CavedonarXiv 2019 · arXiv

TL;DR

This paper introduces SCSdata, a dataset of spoken conversational search interactions, and SCoSAS, the first multi-level annotation schema designed to capture the nuances of audio-only search. The authors demonstrate that spoken search requires a two-tiered interaction model that separates the core search task from discourse management, highlighting the need for future conversational assistants to actively engage in mutual grounding and proactive assistance.

What Problem Does This Paper Address?

While modern voice assistants like Google Home, Amazon Alexa, and Apple HomePod excel at answering simple, factoid questions ("What's the weather today?"), they struggle with complex, multi-turn information-seeking tasks. When a user needs to explore a topic, reformulate vague queries, or parse through dense search results, a purely audio-only channel introduces severe cognitive constraints.

Traditional Information Retrieval (IR) models assume a screen-based interface where users can scan results rapidly. In an audio-only environment, information is delivered linearly, meaning users cannot easily skim. To design effective Spoken Conversational Search (SCS) systems, we first need to understand how humans naturally navigate search tasks using only their voices. This paper addresses the lack of formal interaction models and standardized annotation schemas for multi-turn spoken search.

Methods

Because no existing SCS system can reliably manage complex, multi-turn dialogues, the authors conducted a Natural Dialogue Study (NDS) to simulate an ideal scenario using two human actors:

  • The Seeker: Given a specific "backstory" or information need, the Seeker had to verbally formulate and resolve their search without reading the prompt directly.
  • The Intermediary: Acted as the search engine interface, executing searches on a desktop computer and reading/summarizing results back to the Seeker.

The actors were placed in separate rooms to eliminate facial expressions and visual cues, forcing them to rely entirely on an audio-only channel.

Figure 1: Experimental setup.

To analyze the transcripts, the authors applied Thematic Analysis, a qualitative method used to identify, analyze, and report patterns (themes) within data. They used a six-step inductive process to iteratively condense over 100 initial conversational codes into a structured, two-tiered annotation schema called SCoSAS (Spoken Conversational Search Annotation Schema).

Datasets

The paper utilizes and introduces two primary datasets:

  1. SCSdata (New): Created specifically for this study. It consists of 1,044 conversational turns across 13 pairs of participants tackling 9 distinct search tasks. These tasks spanned three cognitive levels of the Taxonomy of Learning: Remember (e.g., "Where does cinnamon come from?"), Understand (e.g., "How are automobile tires recycled?"), and Analyse (e.g., "How have Turkish river control projects affected Iraqi water resources?").
  2. MISC (Microsoft Information-Seeking Conversation data): An independent, pre-existing dataset of spoken search interactions used to validate the generalizability of the SCoSAS schema. The authors sampled 16 task-instances (666 audio-verified turns) from MISC for cross-dataset evaluation.

Key Insights and Ideas

  • The Two-Tiered Architecture: Spoken search is fundamentally split into the Task Level (actions directly related to the search goal, like query formulation and results presentation) and the Discourse Level (actions maintaining the communication channel, like checking for understanding or navigating pages).
  • Grounding as Implicit Relevance Feedback: Seekers frequently use "grounding" actions—paraphrasing or summarizing what the Intermediary just said to ensure they are on the same page. For an AI system, these grounding moments are goldmines for implicit relevance feedback, revealing exactly how the user interprets the retrieved data.
  • Decision Offloading: Due to the high cognitive load of remembering spoken information, Seekers frequently offload autonomy to the Intermediary (e.g., saying "It's up to you if we look at the other site or not"). This indicates that SCS systems must be designed to be proactive partners rather than passive tools.

Key Results

  • The SCoSAS schema successfully categorized conversational moves into distinct sub-themes (Information Request, Results Presentation, Search Assistance, Search Progression, Discourse Management, Grounding, Navigation, and Visibility of System Status).
  • Annotation Consistency: The schema achieved a moderate Cohen’s $\kappa = 0.59$ at the highly granular atomic code level and a substantial $\kappa = 0.71$ at the broader sub-theme level, proving its reliability for human annotators.
  • Cross-Dataset Validation: When applying SCoSAS to the independent MISC dataset, 94% of the utterances could be successfully coded using the schema. The remaining 6% consisted of setup-specific behaviors like social chitchat or explicit spelling requests, confirming that SCoSAS provides excellent coverage of the conversational search space.

Limitations and Future Work

  • Human-to-Human Bias: The dataset models how humans talk to other humans. The authors acknowledge that users might project different expectations or display altered behaviors when interacting with a machine (the "mindlessness projection" effect). Future work will focus on validating these hypotheses in true human-machine interaction settings.
  • Laboratory Constraints: The study was conducted in a controlled lab environment with pre-defined tasks, which may not fully capture the messy, spontaneous information needs of daily life.
  • Turn-Level Coarseness: The coding schema treats an entire conversational turn as the unit of analysis based on who holds the initiative. This design choice means subtle intra-turn dynamics, such as mid-utterance hesitations or long pauses, were not captured.

Summary updated

Abstract

Conversation is the natural mode for information exchange in daily life, a spoken conversational interaction for search input and output is a logical format for information seeking. However, the conceptualisation of user-system interactions or information exchange in spoken conversational search (SCS) has not been explored. The first step in conceptualising SCS is to understand the conversational moves used in an audio-only communication channel for search. This paper explores conversational actions for the task of search. We define a qualitative methodology for creating conversational datasets, propose analysis protocols, and develop the SCSdata. Furthermore, we use the SCSdata to create the first annotation schema for SCS: the SCoSAS, enabling us to investigate interactivity in SCS. We further establish that SCS needs to incorporate interactivity and pro-activity to overcome the complexity that the information seeking process in an audio-only channel poses. In summary, this exploratory study unpacks the breadth of SCS. Our results highlight the need for integrating discourse in future SCS models and contributes the advancement in the formalisation of SCS models and the design of SCS systems.
BibTeX
@misc{trippas2019towards,
  author = {Johanne R. Trippas and Damiano Spina and Paul Thomas and Mark Sanderson and Hideo Joho and Lawrence Cavedon},
  title = {{Towards a Model for Spoken Conversational Search}},
  year = {2019},
  eprint = {1910.13166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.13166},
}
Figure