TL;DR
This paper tackles the challenge of accurately recognizing and correcting spoken web search queries, especially in noisy mobile environments. They introduce a large, search-specific query corpus and demonstrate that using a custom vocabulary and language model significantly outperforms generic approaches, leading to a practical voice search system with a user-friendly correction interface.
What Problem Does This Paper Address?
Spoken web search queries present unique challenges for automatic speech recognition (ASR). Unlike natural language sentences, search queries are often short, lack grammatical structure, and draw from a vast, diverse vocabulary that includes many proper nouns, slang, and technical terms not found in standard dictionaries. This makes it difficult for traditional ASR systems, trained on general language, to achieve high accuracy. Furthermore, voice search is often performed in noisy, mobile environments, exacerbating recognition errors. The authors aim to build a robust voice search system that can handle these challenges by developing specialized acoustic and language models and incorporating an effective error correction mechanism.
Methods
The core of their approach involves building a specialized ASR system tailored for web search queries. Key methods include:
- Corpus Creation: They compiled a massive corpus of over 10 million web search queries by mining search engine "spy" pages and leveraging Yahoo!'s "related search" feature. This corpus was then augmented with queries generated from seed words found in dictionaries, Wikipedia titles, and Google n-grams.
- Search-Specific Language Models (LMs): They trained LMs specifically on their collected search query corpus. They also explored mixture LMs, combining search data with newswire data, finding that a small amount of newswire data could slightly improve performance.
- Custom Vocabularies: To address the OOV (out-of-vocabulary) problem, they created search-specific vocabularies. For words not present in standard dictionaries (like CMU), they employed a joint multigram model to automatically generate pronunciations using graphones.
- Error Correction Interface: Recognizing that errors are inevitable, they integrated a correction interface based on a word confusion network. This interface presents users with alternative word choices derived from the ASR lattice, allowing for efficient correction without requiring full re-speaking. They also included an on-screen keyboard for more granular corrections.
Datasets
- Search Query Corpus: A newly created corpus of over 10 million web search queries, used for training language models and vocabularies.
- Development Test Set: 755 queries recorded by one of the authors, used for tuning and evaluating the system.
- User Study Audio: 776 queries collected from 4 users performing search tasks while walking, used for offline evaluation and to assess real-world performance.
- CMU Pronouncing Dictionary: A standard dictionary used as a baseline vocabulary.
- WSJ (Wall Street Journal) Corpus: Used for training speaker-independent acoustic models.
- CSR-III Newswire Corpus: Used for training newswire language models.
Key Insights and Ideas
- Search-Specific is Superior: A language model trained on web search queries significantly outperforms one trained on general newswire text, highlighting the unique linguistic characteristics of search.
- Vocabulary Matters: Using a vocabulary that includes words common in web searches, even if pronunciations need to be generated, drastically reduces OOV rates and improves recognition accuracy compared to limiting vocabulary to standard dictionaries.
- Correction is Crucial: Despite high word error rates (WER), users can effectively correct queries using a confusion network interface, making voice search practical even with imperfect recognition.
- Mobile Context is Challenging but Feasible: The study demonstrates that users can speak and correct queries in a noisy, mobile environment within a reasonable timeframe, suggesting the viability of voice search for on-the-go tasks.
Key Results
- Reduced OOV Rates: Using a search-specific vocabulary reduced OOV rates significantly compared to a CMU dictionary-limited vocabulary.
- Improved WER: The best system achieved a WER of 26.3% on the development test set and 35.1% on the user study audio (using a 100K search vocabulary and mixture LM).
- Efficient Correction: The confusion network interface allowed an "oracle" user to correct over half of the recognition errors, reducing WER from 26% to 12%.
- User Study Performance: Users spoke and corrected queries in an average of 18 seconds, despite a 48% WER. Their mean entry rate was 1.7 characters per second, significantly faster than traditional mobile typing.
Limitations and Future Work
- Small User Study: The user study involved only four participants, limiting the generalizability of the findings.
- No Full Search Task: Participants only spoke and corrected queries; they did not perform actual web searches or browse results, which could introduce different error patterns and user behaviors.
- Latency: Users waited an average of 2.7 seconds for recognition results, which could be improved.
- Future Work: The authors suggest that further improvements could be made by optimizing recognition latency and correction interface efficiency. They also envision voice search as a valuable addition to mobile devices, especially with a robust correction mechanism.
Summary updated