TL;DR
This paper tackles the challenge of improving speech-driven search accuracy by tightly integrating Automatic Speech Recognition (ASR) and search components. Instead of treating them as sequential, independent steps, the authors demonstrate that by using search relevance information to re-rank ASR outputs, both ASR and search accuracy can be significantly boosted.
What Problem Does This Paper Address?
Mobile devices, with their limited screen real estate and the prevalence of "hands-busy" scenarios, are increasingly relying on voice interfaces. A common application is speech-driven search, where users speak queries to find information, such as business listings. Traditional approaches treat this as a two-stage process: first, an ASR system converts speech to text, and then a search engine uses this text as a query. However, ASR is prone to errors, and a single "best guess" text output can lead to suboptimal search results. The paper addresses the gap of improving search accuracy by leveraging the search context to refine the ASR output, rather than solely focusing on making the ASR itself perfect in isolation.
Methods
The core innovation lies in the tight coupling of ASR and search. The authors propose representing the ASR output not just as a single best hypothesis, but as a word-confusion network (WCN). A WCN is a compact representation of multiple hypotheses, allowing for flexibility in exploring alternative word sequences.
The key technique is rescoring these WCNs using search relevance metrics. Instead of just accepting the ASR's confidence scores, the paper assigns new weights to words within the WCN based on how relevant they are to the search repository (in this case, a business listing database). This is achieved by:
- Building a Search Index as an FST: The business listings are indexed and represented as a weighted finite-state transducer (SearchFST). Each word in a listing is associated with a weight reflecting its importance.
- Defining Relevance Metrics: Several metrics are explored to quantify word relevance, including variations of inverse document frequency (IDF) and term frequency (TF). The most effective metric found is a combination of term frequency within a document and inverse document frequency across the collection.
- Composing ASR Output with SearchFST: The WCN (representing ASR hypotheses) is composed with the SearchFST. This process effectively re-ranks the paths within the WCN based on search relevance.
- Pruning and Selecting Top Listings: The re-ranked WCN is then used to query the search engine, and the top N business listings are returned.
Datasets
- Business Listing Database: A database of over 10 million business listings (containing name, address, and phone number) was used as the information repository.
- Text Query Logs: Over 10 million text query logs from a web-based text-driven search application were used to build the ASR language model.
- Speech Queries: 852 speech queries collected from users of a mobile device speech search application were used for evaluation. These were split into a development set (300 utterances) and a test set (552 utterances).
Key Insights and Ideas
- The "aha moment" is that ASR errors are not uniformly bad for search. Some errors might be less detrimental if the words involved are highly relevant to the search context. Conversely, even a "correctly" recognized word might be irrelevant to the search task.
- Tightly coupling ASR and Search is more effective than sequential processing. By allowing search relevance to influence ASR output interpretation, we can achieve better results for both components.
- Word-Confusion Networks (WCNs) are a powerful intermediate representation. They compactly encode multiple ASR hypotheses, enabling sophisticated re-ranking strategies.
- Search relevance metrics can significantly improve ASR accuracy. Even a small percentage improvement in ASR accuracy (e.g., 1% word-level) can lead to a substantial boost in search accuracy.
Key Results
- ASR Accuracy Improvement: The best rescoring metric improved word-level ASR accuracy from 63.1% to 63.9% on the development set and from 65.1% to 65.9% on the test set. Sentence-level accuracy improved from 57.0% to 58.3% (dev) and 55.3% to 56.2% (test).
- Search Accuracy Improvement: The F-score for search results improved from 0.718 (baseline ASR output) to 0.735 (reranked ASR output) on the test set. Precision and Recall also saw notable improvements.
- Oracle Accuracy: The paper shows that by increasing the arc density of the WCN, oracle accuracies (the best possible path in the WCN) can be improved by almost 10% absolute over the 1-best accuracy.
Limitations and Future Work
- Approximation of Reference Search Set: The paper acknowledges that using the SearchFST output for human transcribed queries as the "reference search set" is an approximation. A truly human-annotated search set would be ideal.
- Future Work: The authors plan to explore other relevance metrics that can capture more sophisticated constraints, such as the relative coherence of terms within a query. They also aim to investigate how to encode these constraints more effectively within the FST framework.
Summary updated