TL;DR
This paper tackles the challenge of improving voice search accuracy by personalizing the results. They introduce a novel log-linear classification framework that leverages short-term, long-term, and web-based user history features to re-rank the initial hypotheses from a speech recognition system, leading to significant reductions in sentence error rates.
What Problem Does This Paper Address?
Voice search applications offer a convenient way to access information, but they are often plagued by speech recognition errors. These errors can lead to incorrect search results, frustrating users and diminishing the utility of the system. While recommender systems have been successful in personalizing text-based search, adapting these techniques to the unique challenges of voice search, particularly the dynamic nature of spoken queries and the immediate feedback loop of recognition errors, remains an open problem. This paper aims to bridge this gap by exploring how to effectively personalize voice search results using various aspects of user behavior.
Methods
The core of the proposed solution is a log-linear classifier used for re-scoring the n-best lists generated by a speech recognition system. This approach, similar to maximum entropy models, allows for the integration of diverse features to predict the most likely correct word sequence.
The novelty lies in the feature extraction process, which draws from three distinct personalization strategies:
- Short-term Personalization: This focuses on modeling user "repair" behavior. If a user repeats or reformulates a query, it often indicates a previous recognition error. Features are extracted based on whether a hypothesis has been seen in recent (e.g., 60-second window) previous n-best lists and whether it was clicked.
- Long-term Personalization: This leverages a user's historical interactions. Features are derived from how often a hypothesis (or similar ones) has appeared in the user's past search history, whether it was the most clicked, or if it was the most recently clicked. This also includes features like edit distance to past queries and common words.
- Live Search-based Features: This approach utilizes external web search data. Features are generated by querying Live Search for each hypothesis and considering its absolute count in web documents and its rank within the search results. This taps into the assumption that if a business exists, it will likely appear in indexed web documents.
Datasets
The experiments were conducted on a dataset of 22,473 transcribed business utterances extracted from a commercially deployed large vocabulary directory assistance system. For evaluation, a subset of 9,858 utterances was created, containing only those where the correct hypothesis was present in the n-best list and there was more than one hypothesis. This subset was chronologically partitioned into training and testing sets.
Key Insights and Ideas
- Multi-scale personalization is key: Combining short-term, long-term, and web-based features yields significantly better results than any single approach. This highlights that different aspects of user behavior provide complementary information.
- Short-term repetition is a strong signal: Even simple features indicating a hypothesis was seen in a recent n-best list can help reduce errors, especially when combined with click information.
- Web search data is a powerful proxy: Features derived from Live Search, particularly those related to relative relevance (how a hypothesis's count compares to others in the n-best list), are highly effective. This suggests that external knowledge about the likelihood of a query's result is valuable.
- User history is more than just occurrences: Features like "user history (alone)" and "user history (most clicked)" provide more nuanced signals than simply counting past appearances.
Key Results
The proposed personalization features significantly reduce the Sentence Error Rate (SER) on the rescoring subset.
- The baseline system (hypothesis rank only) achieved an SER of 24.73%.
- Incorporating short-term personalization features reduced the SER to 24.32%.
- Long-term personalization features, particularly "UH (occurrences)," reduced the SER to 23.76%.
- Live Search (relative I) feature alone reduced the SER to 23.51%.
- The combination of all proposed features achieved the lowest SER of 21.54%, representing an absolute reduction of 3.19% on the rescoring set and 1.40% on the entire dataset.
Limitations and Future Work
- Inconsistencies in user interaction: The paper notes that inconsistencies in user interaction (e.g., a correct listing not being confirmed by the user) may have prevented further improvement with short-term features.
- Feature design is critical: The authors emphasize that the "right design of the features seems critical to take full advantage" of personalization, particularly for web search data.
- Future work could explore: While not explicitly detailed, the success of combining diverse features suggests that further exploration of more sophisticated feature engineering and potentially different modeling approaches could yield even greater improvements. The paper also hints at the potential of personalized pronunciations, which was not the focus of this particular study.
Summary updated