Toward Connecting Speech Acts and Search Actions in Conversational Search Tasks

Souvick Ghosh, Satanu Ghosh, Chirag Shah

Abstract

Modern conversational assistants excel at simple, factoid-based queries but often struggle with the complex, multi-turn interactions required for exploratory information seeking. This research addresses the disconnect between user intent and system execution by proposing a framework that maps "speech acts" (the functional intent of a user's utterance) to "search actions" (the specific operations performed by a search engine). Using a multi-channel Attention-based Deep Neural Network (ADNN) and a newly developed dataset, CONVEX, this study demonstrates that combining transformer-based embeddings with linguistic features and conversational metadata significantly improves the prediction of system-level actions in conversational search.

Introduction

While voice-activated assistants like Siri and Alexa have become ubiquitous, their utility is largely confined to "factoid" queries (e.g., "What is the height of the Eiffel Tower?") or simple command executions. However, human information seeking is often non-factoid and exploratory, resembling the nuanced back-and-forth dialogue found in a reference library. In these scenarios, a single query is rarely sufficient. Instead, users engage in a discourse where their needs evolve through clarification and refinement.

This paper identifies a critical gap in current conversational systems: the lack of a formal connection between the natural language used by humans and the technical actions a search system must perform. To bridge this gap, the researchers propose a model that identifies the functional aspects of discourse—speech acts—and uses them to trigger appropriate system-level search actions, such as query creation or document scanning.

Methodology

Data Collection: The Wizard-of-Oz Study

To capture authentic human-system interaction, the researchers conducted a Wizard-of-Oz (WoZ) study involving 25 users across 75 sessions. Participants interacted with what they believed was an AI, but was actually a human "Wizard"—an experienced searcher following strict protocols. This approach allowed for the collection of realistic dialogue data in a controlled environment.

Thematic Analysis and Annotation

The resulting dialogues were manually annotated using the Conversational Roles (COR) model. The researchers identified:

  • 12 Speech Acts: Including categories such as Question/Seek, Clarify, and Repeat.
  • 4 Search Actions: Including Query Creation, SERP (Search Engine Results Page) Scanning, and Document Scanning.

The ADNN Model

The core of the proposed solution is an Attention-based Deep Neural Network (ADNN). This model utilizes a multi-channel architecture to process three distinct types of input simultaneously:

  1. Word-level features: Leveraging DistilBERT for efficient natural language understanding.
  2. Linguistic features: Utilizing the SpaCy API to extract parts-of-speech and syntactic dependencies.
  3. Metadata: Incorporating contextual information such as the utterance number, the speaker's role, and the previous speech act.

Experimental Setup and Datasets

The study utilized two primary datasets for training and validation:

  • CONVEX (CONversation with EXplanations): A new "gold standard" dataset created by the authors, containing 1,834 utterances and 509 system actions derived from the WoZ study.
  • SCS (Spoken Conversational Search): An existing, publicly available dataset used to test the generalizability of the models across different conversational contexts.

Key Results and Discussion

Classification Performance

The multi-channel model demonstrated strong performance in identifying user intent but faced greater challenges in predicting system actions:

  • Speech Act Classification: The integrated model (Meta + Linguistic + BERT) achieved a maximum accuracy of 90.2% on the CONVEX dataset and 72.7% on the SCS dataset.
  • Search Action Classification: Accuracy was notably lower, peaking at 58.8% on CONVEX and 61.1% on SCS.

The Role of Context

A significant finding of the study was the importance of metadata. Ablation studies revealed that removing conversational history (the previous speech act) caused the most substantial drop in performance for search action prediction. This suggests that while the current utterance tells the system what is being said, the conversational context tells the system why it is being said and what should happen next.

Multi-Channel Synergy

The research highlights that while transformer models like BERT are powerful, they are not a panacea for specialized domains. The best results consistently came from combining BERT with "hand-engineered" linguistic and contextual features. This synergy is essential for understanding the nuances of search-related discourse.

Limitations and Future Directions

The primary hurdle identified by the authors is data scarcity; while CONVEX is a valuable contribution, deep learning models typically require larger datasets to achieve high accuracy in complex tasks like search action prediction. Additionally, class imbalance—where certain speech acts occur much more frequently than others—remains a challenge.

Future work will focus on data augmentation techniques, such as SMOTE, to balance the dataset. The ultimate goal is to integrate this pipeline into a live, proactive conversational agent capable of recognizing topic changes and offering intelligent query modifications in real-time digital library environments.

Conclusion

This research provides a foundational framework for moving beyond reactive conversational assistants. By formalizing the link between speech acts and search actions, the study paves the way for systems that can truly partner with users in complex information-seeking tasks, understanding not just the words spoken, but the functional intent behind them.

Summary updated

Abstract

Conversational search systems can improve user experience in digital libraries by facilitating a natural and intuitive way to interact with library content. However, most conversational search systems are limited to performing simple tasks and controlling smart devices. Therefore, there is a need for systems that can accurately understand the user's information requirements and perform the appropriate search activity. Prior research on intelligent systems suggested that it is possible to comprehend the functional aspect of discourse (search intent) by identifying the speech acts in user dialogues. In this work, we automatically identify the speech acts associated with spoken utterances and use them to predict the system-level search actions. First, we conducted a Wizard-of-Oz study to collect data from 75 search sessions. We performed thematic analysis to curate a gold standard dataset - containing 1,834 utterances and 509 system actions - of human-system interactions in three information-seeking scenarios. Next, we developed attention-based deep neural networks to understand natural language and predict speech acts. Then, the speech acts were fed to the model to predict the corresponding system-level search actions. We also annotated a second dataset to validate our results. For the two datasets, the best-performing classification model achieved maximum accuracy of 90.2% and 72.7% for speech act classification and 58.8% and 61.1 %, respectively, for search act classification.
BibTeX
@inproceedings{ghosh2023toward,
  author = {Souvick Ghosh and Satanu Ghosh and Chirag Shah},
  title = {{Toward Connecting Speech Acts and Search Actions in Conversational Search Tasks}},
  booktitle = {JCDL},
  year = {2023},
  url = {https://openreview.net/forum?id=tQfbstqeAC},
}
Figure