Prosody Modifications for Question-Answering in Voice-Only Settings

Aleksandr Chuklin, Aliaksei Severyn, Johanne R. Trippas, Enrique Alfonseca, Hanna Silén, Damiano Spina

Introduction

As digital interaction shifts from screen-based search to voice-only assistants like Alexa or Google Home, a significant user experience challenge emerges: the loss of visual cues. On a screen, search engines can bold an "answer" within a text snippet, allowing users to skim and identify relevant information instantly. In a voice-only setting, communication is serial; users must listen to the entire response, placing a heavy burden on short-term memory and cognitive processing. This paper explores how modifying speech synthesis—using pauses, pitch shifts, and rate changes—can act as "audio bolding" to help users identify key information more effectively.

Example of a question and answer sentence with a highlighted answer key used in the study.

Methodology

The researchers proposed a framework to evaluate "prosodic modifications" using two Text-to-Speech (TTS) engines: IBM Watson and Google Wavenet. They tested four specific modifications applied to the "answer key" within a full sentence, derived from the SQuAD (Stanford Question Answering Dataset):

  1. Pause: Inserting silence before and after the key phrase.
  2. Rate: Slowing down the speaking speed for the key phrase.
  3. Pitch: Raising the pitch of the key phrase.
  4. Emphasis: A combination of increased pitch and slower rate (utilizing Google’s built-in <emphasis> tag).

Evaluation was conducted via crowdsourcing, where workers provided feedback on informativeness (answering quality), elocution (pronunciation), interruption (unwanted breaks), and correctness (the ability to accurately identify and transcribe the highlighted answer).

Key Results and Insights

The study treats prosody not merely as a tool for naturalness, but as a functional UI element akin to CSS styling. The results indicate that users are willing to trade some naturalness for increased utility.

  • The "Emphasis" Advantage: Google’s emphasis tag, which blends pitch and rate changes, was the most effective modification. It showed a +0.87 increase in informativeness and a +0.28 increase in correctness over standard speech baselines.
  • Context and Length Sensitivity: Highlighting techniques are context-dependent. For instance, pauses significantly improved informativeness for short answers (+1.09) but were perceived as unnatural and frustrating for longer phrases, where informativeness dropped by -0.97.
  • Positional Effects: Modifications were most effective when the answer appeared toward the end of the sentence, likely because it reduced the cognitive load required to retain the answer until the audio finished.
Comparison of informativeness and correctness distributions between baseline speech and emphasis modifications.

Discussion and Future Work

While prosodic modifications often increased "interruption" scores (making the voice sound less natural), users consistently rated them as more useful for information digestion. This suggests that for task-oriented voice interactions, functional clarity can outweigh human-like prosody.

Future research should focus on optimizing the "strength" of these modifications dynamically based on answer length or sentence complexity. Additionally, as this study was limited to English, further investigation is required to determine how "audio bolding" rules translate to tonal languages or different linguistic contexts where prosodic cues carry different meanings.

Summary updated

Abstract

Many popular form factors of digital assistants---such as Amazon Echo, Apple Homepod, or Google Home---enable the user to hold a conversation with these systems based only on the speech modality. The lack of a screen presents unique challenges. To satisfy the information need of a user, the presentation of the answer needs to be optimized for such voice-only interactions. In this paper, we propose a task of evaluating the usefulness of audio transformations (i.e., prosodic modifications) for voice-only question answering. We introduce a crowdsourcing setup where we evaluate the quality of our proposed modifications along multiple dimensions corresponding to the informativeness, naturalness, and ability of the user to identify key parts of the answer. We offer a set of prosodic modifications that highlight potentially important parts of the answer using various acoustic cues. Our experiments show that some of these prosodic modifications lead to better comprehension at the expense of only slightly degraded naturalness of the audio.
BibTeX
@inproceedings{chuklin2019prosody,
  author = {Aleksandr Chuklin and Aliaksei Severyn and Johanne R. Trippas and Enrique Alfonseca and Hanna Silén and Damiano Spina},
  title = {{Prosody Modifications for Question-Answering in Voice-Only Settings}},
  booktitle = {CLEF},
  year = {2019},
  doi = {10.1007/978-3-030-28577-7\_12},
}
Figure