TL;DR
This paper introduces a hybrid architecture for mobile voice user interfaces that combines a powerful network-based speech recognizer with a compact on-device recognizer. This approach significantly speeds up response times for common voice commands by handling them locally, while still leveraging the network for more complex queries, leading to a better user experience without a substantial drop in accuracy.
What Problem Does This Paper Address?
Mobile voice user interfaces (VUIs) are becoming increasingly popular, but traditional network-only Automatic Speech Recognition (ASR) systems suffer from latency issues. This delay, caused by wireless connections and server load, can make VUIs feel sluggish and unresponsive, especially for quick commands like voice dialing or launching apps. The paper aims to solve this by intelligently distributing the ASR workload between the mobile device and the network, prioritizing local processing for predictable queries to minimize latency while maintaining overall system robustness.
Methods
The core of the proposed solution is a hybrid ASR architecture. It features:
- Network-side Large Vocabulary Continuous Speech Recognizer (LV-CSR): This is the standard, powerful ASR engine that handles all types of queries. It also performs online adaptation based on device and application IDs.
- On-device Compact Embedded Recognizer: This smaller, faster recognizer is designed to handle a subset of common queries. Its vocabulary is dynamically configured.
The key innovation lies in the Vocabulary Prediction (VP) mechanism used to populate the on-device recognizer's vocabulary. This VP process combines three sources of information:
- Most Commonly Used Sentences (Top N): A list of the most frequent utterances from the live ASR service.
- Sentence Templates: Rule-based templates for predictable sentence structures (e.g., "call NAME"). These are instantiated with data from the user's contact list and application list.
- Usage History: Sentences previously used by the specific user.
A scoring algorithm combines these sources to create the local vocabulary. When a user speaks, both recognizers are activated simultaneously. If the on-device recognizer successfully recognizes the utterance (i.e., it's not flagged as Out-Of-Vocabulary or OOV), the network recognition is canceled, and the result is returned instantly. If it's OOV, the system waits for the network recognizer's response.
Datasets
The paper utilizes field data from Vlingo's live mobile ASR service across six languages: US English (en-us), UK English (en-uk), Finnish English (en-fi), German (de-de), Italian (it-it), and Spanish (es-es). For experimental purposes, they selected 100 users per language, each with approximately 200 sentences of interaction history. The last 50 sentences of each user's history were used as the test set, and the preceding 150 sentences formed the training set for user history. Contact lists from these users were also used.
Key Insights and Ideas
- Intelligent Distribution is Key: The hybrid approach demonstrates that a significant portion of voice queries can be handled locally without sacrificing accuracy, drastically improving user experience.
- User Data is Gold: Leveraging user-specific data (contacts, usage history) is crucial for effective vocabulary prediction and maximizing on-device recognition.
- Templates Unlock Predictability: Sentence templates, especially for PIM tasks, are highly effective for generating local vocabulary, as they capture common, structured utterances.
- Trade-offs are Manageable: The paper shows that a small increase in overall word error rate (less than 5%) can yield substantial gains in on-device recognition (28-55%).
Key Results
- On-device Recognition Rates: Depending on the availability of user history, the system can handle 28-55% of voice queries locally.
- Error Rate Impact: This local processing comes at the cost of a less than 5% relative increase in the overall word error rate.
- Response Time: For locally handled utterances, recognition is virtually instantaneous, with algorithmic delays less than 100ms.
- Language Variability: Performance varies across languages, with some (like Italian, Spanish, Finnish English) showing better on-device figures, potentially due to more predictable sentence usage.
- Memory Footprint: The local recognizer is very small, operating in less than 1MB of RAM even with a vocabulary of 5000 sentences.
- Speed: The local recognizer operates much faster than real-time, with real-time factors (xRT) between 0.35-0.39 on a Nokia E71 and 0.17-0.22 on a Nokia N97.
Limitations and Future Work
- Contact Data Quality: The paper acknowledges significant challenges with user contact databases, including cryptic entries, typos, and multiple identical entries, which require cleanup.
- Preloaded Contacts: The variability in preloaded contacts across countries and the nomenclature used for services present an area for further investigation.
- Acoustic Adaptation: While promising, the experimental results for acoustic adaptation were based on supervised learning and large datasets, which are not entirely realistic for practical deployment.
- Future Work: The authors suggest exploring more advanced embedded speech recognizers capable of handling statistical language models locally, which could further relax vocabulary prediction criteria and improve local coverage. They also plan to investigate the system's performance with real field deployment data.
Summary updated