TL;DR
This paper explores the use of non-lexical audio cues (like beeps, prosody changes, and speaker shifts) in speech interfaces to convey information beyond spoken words. They found that natural prosody is surprisingly effective for indicating system confusion and list continuation in limited-domain systems, while speaker switching aids lexical entrainment.
What Problem Does This Paper Address?
Speech interfaces, especially those with limited or no visual displays, often struggle to convey nuanced information efficiently. This can range from indicating system errors or the end of a list to guiding users on how to interact. The authors aim to investigate how non-lexical audio signals – sounds and supra-lexical features that aren't part of the core spoken message – can be used to enhance the user experience in these interfaces. They are particularly interested in whether these signals have intuitive meanings or if they need to be learned, and how they perform in different communicative contexts within a Universal Speech Interface (USI) framework.
Methods
The researchers created a set of audio utterances designed to test various non-lexical cues in specific scenarios. These scenarios included: system confusion, list continuation/finality, feature specification (like value types and optionality), and lexical entrainment. For each scenario, they employed several types of non-lexical cues:
- Prosodic Variation: Altering pitch, power, or speed of spoken words.
- Inserted Noises: Adding beeps or other distinct sounds.
- Background Noises: Playing sounds continuously during an utterance.
- Speaker Switching: Changing to a different voice (e.g., male vs. female) or synthesis style for specific parts of the utterance.
They used a limited-domain speech synthesis system (Festival) and also recorded a human speaker. Some utterances were created using diphone synthesis, while others involved modifying recordings of the human speaker.
The evaluation involved two parts:
- Free-form Interpretation: Users listened to individual utterances without context and described what information they received.
- Comparative Assessment: Users were presented with multiple options for a given scenario and asked to choose the one that best conveyed the intended meaning.
The study surveyed 14 participants, most of whom were unfamiliar with the USI project.
Datasets
The paper does not explicitly mention pre-existing datasets for training. Instead, they created their own set of audio stimuli for evaluation. This set consisted of 33 utterances, which can be found at http://www.speech.cs.cmu.edu/usi/audiomarking.htm. This collection serves as a benchmark for evaluating non-lexical cues in the specific contexts studied.
Key Insights and Ideas
- Natural Prosody is Powerful (in Limited Domains): For conveying system confusion and list continuation, natural prosody (like rising intonation for questions or a non-falling tone at the end of a list) was surprisingly effective. This suggests that even subtle, human-like variations can be intuitively understood.
- The Challenge of Diphone Synthesis: Users found diphone-synthesized speech often unnatural and unintelligible. This made it difficult for them to perceive or interpret any imposed prosodic variations, suggesting that such variations are best applied to more natural-sounding speech.
- Arbitrary Noises Need Careful Selection: While beeps and other artificial noises were more noticeable than prosody in some cases, their effectiveness depends heavily on avoiding pre-existing meanings. A beep that sounds like a phone's call-waiting signal, for instance, can cause confusion rather than convey information.
- Speaker Switching for Entrainment: Changing the speaker for specific phrases (e.g., from a system operator to a model user) proved effective in encouraging users to adopt the correct command structure (lexical entrainment).
- Optionality is Tricky: Conveying optionality non-lexically was challenging, with participants struggling to discern the intended meaning from the tested cues. This highlights a potential area for future research.
Key Results
- System Confusion: Natural prosody was the most popular choice for indicating system confusion. A steady background tone was the second most popular, though not significantly better than other options.
- List Continuation/Finality: Natural prosody was again preferred for indicating the continuation or end of a list. A quick three-beep signal was the next most popular option.
- Feature Specification: For value types (like time or amount), non-arbitrary noises (clock ticking for time, cash register for amount) were preferred when they replaced the spoken value. However, for indicating optionality, no clear preference emerged among the tested cues.
- Lexical Entrainment: Switching speakers for specific phrases was preferred by a significant margin for enhancing lexical entrainment.
Limitations and Future Work
- Limited Domain Dependency: The effectiveness of natural prosody is strongly tied to limited-domain synthesis. For larger, more open-ended domains, achieving this level of nuanced prosody automatically is difficult.
- Diphone Synthesis Issues: The unintelligibility of diphone synthesis hindered the perception of non-lexical cues. This suggests that if diphone synthesis is necessary, learned, arbitrary noises might be a more reliable approach for conveying information.
- Arbitrary Noise Learnability: The paper acknowledges that while inserted noises are easier to implement, their learnability (how easily users pick up their meaning) needs further investigation.
- Optionality Conveyance: The difficulty in conveying optionality non-lexically suggests this is an area ripe for future research.
- Contextual vs. Non-Contextual Meaning: While the initial tests explored non-contextual meanings, the authors note that the full potential of these cues might be realized when integrated within a richer interaction context.
Summary updated