TL;DR
This paper introduces "Speech Graffiti," a structured, universal speech interface protocol designed to overcome the challenges of natural language speech interfaces. A user study found that participants overwhelmingly preferred Speech Graffiti over a natural language interface for a movie information task, citing higher satisfaction and comparable task completion rates, despite a higher number of turns.
What Problem Does This Paper Address?
Speech-based interfaces hold immense promise, but their widespread adoption is hindered by inherent difficulties with spoken language: variability in pronunciation, noisy environments, and ambiguity in user intent. While natural language interfaces offer flexibility, they can be inefficient and difficult for novice users. Conversely, specialized command-and-control languages offer power but require learning a new system for each application. This paper explores a middle ground: a structured, universal speech interface protocol that aims to be both efficient and easy to learn across different applications. The core question is whether users would prefer such a structured, potentially more efficient, interface over a more natural but potentially less efficient one.
Methods
The researchers conducted a within-subjects user study involving 23 participants. Each participant interacted with two different movie information systems: one using a Speech Graffiti interface (SG-ML) and another using a natural language interface (NL-ML). The order of system presentation was counterbalanced.
Participants first underwent a self-paced, web-based tutorial for Speech Graffiti, which included examples from both a movie database and a simulated flight information system to familiarize them with the universal nature of the protocol. For the NL-ML, a brief "familiarization session" was provided to balance the effect of learning a new interface, with half of the participants using a natural language movie system and the other half a natural language weather system.
After training, participants were tasked with completing a set of eight queries on each system. Following interaction with each system, participants completed a 34-item subjective satisfaction questionnaire (based on the SASSI project) and were asked to state their preference between the two interfaces. Objective measures included task completion rates, time-to-completion, turns-to-completion, word-error rate (WER), and an initial assessment of understanding error rate (UER).
Datasets
The study used a movie information database for the primary task. For training and familiarization, examples from a simulated flight arrival, departure, and gate information system, and MIT's Jupiter natural language weather information system were also utilized. No new datasets were introduced.
Key Insights and Ideas
- Structured Interfaces Can Be Preferred: The most striking finding is that users overwhelmingly preferred the structured Speech Graffiti interface over the natural language one, suggesting that efficiency and learnability can outweigh the perceived "naturalness" of an interface.
- Trade-off Between Turns and Cognitive Load: Speech Graffiti's higher turn-to-completion rate, while seemingly less efficient, might be a trade-off for reduced cognitive load. The structured output, broken into smaller chunks, may require more turns but makes processing easier for the user.
- WER is a Strong Predictor of Preference: Users who preferred Speech Graffiti generally had lower word-error rates with that system. Conversely, the few users who preferred the natural language interface also exhibited the highest error rates with Speech Graffiti, highlighting the critical role of accurate speech recognition in user acceptance.
- Understanding Error is Crucial: The paper introduces the concept of Understanding Error Rate (UER) as potentially more indicative of system performance than WER, especially for natural language systems where grammatical redundancies can allow for correct understanding despite word errors. Speech Graffiti's structured nature inherently reduces the complexity of matching decoded words to concepts, thus mitigating understanding errors.
Key Results
- Preference: 74% (17 out of 23) of participants preferred the Speech Graffiti system.
- Subjective Satisfaction: Speech Graffiti received significantly higher user satisfaction scores across all six measured factors (system response accuracy, habitability, cognitive demand, annoyance, likeability, and speed) and overall.
- Task Completion: Task completion rates were nearly identical for both interfaces (67.4% for NL-ML, 67.9% for SG-ML).
- Time-to-Completion: For completed tasks, SG-ML was slightly faster (67.9 seconds vs. 71.3 seconds). When including incomplete tasks, SG-ML showed a more significant advantage with a median time of 81.5 seconds compared to 103 seconds for NL-ML.
- Turns-to-Completion: SG-ML required significantly more turns per task (median 10 vs. 5 for NL-ML).
- Word-Error Rate (WER): SG-ML had a substantially lower overall WER (35.1%) compared to NL-ML (51.2%).
Limitations and Future Work
- Habitability: The study identified "habitability" (users knowing what to say) as a key area for improvement in Speech Graffiti, particularly for users who struggled with the structured grammar.
- User Group: The study participants were primarily undergraduate students, limiting the generalizability of findings to broader adult populations.
- Future Directions: The authors plan to explore more natural language-esque interaction within the Speech Graffiti framework while maintaining its application-portable structure. They also aim to enhance runtime help facilities to better assist users. Furthermore, they intend to extend Speech Graffiti's functionality beyond information access to support data creation, deletion, and modification. The study also motivates future work on helping users who have severe difficulties with an interface learn it more effectively and faster.
Summary updated