Examining Prosody in Spoken Navigation Instructions for People with Disabilities

Cathy Jiao, Aaron Steinfeld, Maxine EskenaziNAACL 2024

TL;DR

This paper investigates how different speech prosody features in spoken navigation instructions affect information recall, particularly for people with disabilities. They found that slowing down speech rate significantly improves number recall, while pauses before keywords are less effective and can even hinder recall. This work offers practical guidance for designing more accessible voice assistants.

What Problem Does This Paper Address?

As voice assistants like Siri and Alexa become ubiquitous, ensuring they are usable by everyone, including people with disabilities, is crucial. While Natural Language Processing (NLP) research has focused on improving the underlying models, user-facing aspects like synthesized speech prosody have received less attention in the context of accessibility. Previous work has shown that prosody can aid comprehension and recall, but it's unclear which specific prosodic features are most beneficial for different types of information (e.g., numbers vs. street names) and for users with varying needs. This paper aims to bridge this gap by evaluating the impact of specific prosodic modifications on the recall of navigation instructions, a task that requires remembering both numerical and textual information.

Methods

The researchers conducted a study where 198 participants (80 with self-reported disabilities and 118 without) listened to navigation instructions generated by the "GetGoing" system. GetGoing uses Google Maps API for directions and Vonage's text-to-speech API for audio output.

They created four distinct audio conditions for the instructions:

  • Default: The standard Vonage API voice.
  • Default-slow: The default voice with a reduced speech rate.
  • Break-short: The default voice with a 5ms pause before every noun and numeric entity.
  • Break-long: The default voice with a 15ms pause before every noun and numeric entity.

Participants were asked to transcribe the instructions they heard, and their recall accuracy for both noun entities (street names, building names) and numeric entities (times, numbers) was analyzed. They also examined how noun and number length, as well as participant age, influenced recall.

Datasets

The study used navigation instructions generated by the "GetGoing" system, which were previously deployed in Southwestern Pennsylvania. The researchers selected 48 unique instructions, balanced across eight parameter groups defined by the number of noun and numeric entities. These instructions were then used to create 192 audio clips (48 instructions x 4 conditions). No new benchmark datasets were introduced.

Key Insights and Ideas

  • Slowing Down is Key for Numbers: The most significant finding is that a slower speech rate (Default-slow) consistently improves the recall of numbers, even as the number of numerical entities in an instruction increases. This suggests that for numerical information, clarity and a more deliberate pace are paramount.
  • Pauses Aren't Always Helpful: Surprisingly, inserting pauses before keywords (Break-short and Break-long) did not improve information recall and, in some cases, led to worse performance, particularly for number recall. This indicates that while prosody matters, simply adding breaks isn't a universal solution and might even be detrimental by creating cognitive overload.
  • Noun Recall is More Robust (to these changes): Noun recall (street names, etc.) was less affected by the tested prosodic variations compared to number recall. While noun accuracy did decrease with longer nouns, the tested prosodic conditions didn't show a strong differential impact.
  • Accessibility is Not One-Size-Fits-All: The study highlights that different types of information might require different prosodic treatments. What benefits number recall might not benefit noun recall, suggesting a need for more nuanced approaches.
  • Online Study Environment: The researchers acknowledge that conducting the study online might have mitigated some accessibility challenges (e.g., participants using headphones or adjusting volume), potentially masking some differences that might appear in less optimal listening environments.

Key Results

  • Number Recall: The Default-slow condition significantly improved number recall compared to the Default condition, especially when instructions contained multiple numbers. The Break-short and Break-long conditions generally performed worse than Default-slow and Default for number recall.
  • Noun Recall: Noun recall accuracy was less sensitive to the tested prosodic conditions. While accuracy decreased for longer nouns (≥ 16 characters), the differences across conditions were not as pronounced as with numbers.
  • Disability and Age: The study found little difference in recall accuracy between participants with and without self-reported disabilities. Similarly, while noun recall showed some fluctuations with age, number recall generally decayed with increasing age, but the prosodic conditions' impact was relatively consistent across age groups, with Default-slow still showing an advantage for numbers.

Limitations and Future Work

The authors acknowledge several limitations:

  • Online Environment: As mentioned, the controlled online environment might not reflect real-world usage, especially for individuals with hearing impairments who might rely on assistive devices or specific environmental setups. Future work should explore less optimal conditions.
  • Limited Prosodic Features: The study focused on speech rate and pauses. Other prosodic features (e.g., pitch, intonation) were not explored.
  • Independent Parameters: The analysis treated noun and number recall as independent. In reality, these elements are often intertwined in navigation instructions.
  • Limited Information Types: The study focused solely on noun and number recall. Navigation instructions often include other critical information like actions ("turn right"), landmarks, or temporal cues.

Future research directions include:

  • Prioritizing Prosodic Cues: Investigating whether applying prosodic cues selectively to high-priority information (e.g., long numbers, uncommon nouns) could be more effective than applying them broadly.
  • Personalization: Exploring how prosodic features can be personalized to individual user preferences and needs, especially for users with disabilities.
  • More Complex Instructions: Extending the study to more complex, multi-step navigation instructions that include a wider range of information types.
  • Real-World Environments: Testing these prosodic features in more realistic and less controlled listening environments.

Summary updated

BibTeX
@inproceedings{jiao2024examining,
  author = {Cathy Jiao and Aaron Steinfeld and Maxine Eskenazi},
  title = {{Examining Prosody in Spoken Navigation Instructions for People with Disabilities}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure