Accommodating explicit user expressions of uncertainty in voice search or something like that

Tim Paek, Yun-Cheng JuInterspeech 2008

TL;DR

This paper tackles the problem of voice search systems failing when users don't know the exact query, proposing a solution where users can explicitly express uncertainty using "something." This approach significantly improves retrieval accuracy compared to guessing, especially in multimodal interfaces, by allowing systems to leverage partial user knowledge.

What Problem Does This Paper Address?

Voice search applications, like automated directory assistance (ADA), aim for users to simply "say what you want." However, a common real-world scenario is users only remembering parts of a name or being slightly mistaken (e.g., "Le Sol Spa" instead of "Le Soleil Tanning and Spa"). Current systems struggle with this, forcing users to guess or rephrase, leading to frustration and failed searches. This paper addresses the gap of explicitly accommodating user uncertainty in voice queries, enabling systems to better understand and act upon partial or imprecise information.

Methods

The authors propose two main methodological advancements:

  • Language Modeling: To handle "something" as a wildcard, they artificially augmented their training data. For existing listings, they created new variations by replacing words with "" or "i-" (initial letter followed by ""). This heuristic approach aimed to improve the recognition of uncertain phrases without overly impacting performance on exact matches. They also modified the pronunciation dictionary to recognize "" as "something."
  • Information Retrieval: They developed a two-pronged approach for retrieving results based on uncertain queries:
  • Exact Matches (K-Best Suffix Array): They encoded directory listings into k-best suffix arrays, allowing for efficient retrieval of popular exact matches for regular expressions derived from the recognized uncertain queries.
  • Approximate Matches (TFIDF): An improved TFIDF algorithm was used to find approximate matches, which is robust to word order errors or spurious words that might arise from speech recognition inaccuracies.

Datasets

  • Training Data: 1.7 million training sentences from Tellme (a Microsoft subsidiary) were analyzed for user uncertainty.
  • Evaluation Data: 2317 transcribed ADA utterances from Microsoft Live Search Mobile were used to evaluate the impact of the proposed language model on overall ADA performance.
  • User Experiment Data: 15 subjects generated 178 utterances, comprising 54 guesses and 54 "something" expressions, for directory listings.

Key Insights and Ideas

  • User Uncertainty is Common: A significant portion (72%) of training sentences in ADA did not match exact listings, and even a conservative estimate found 0.5% of utterances contained "something" as an expression of uncertainty. This suggests a substantial user need for handling such queries.
  • "Something" as a Powerful Wildcard: Explicitly allowing users to say "something" for unknown parts of a query dramatically improves retrieval accuracy compared to random guessing.
  • Multimodal Interfaces Benefit Most: While uncertain expressions improve accuracy across the board, the benefit is more pronounced in multimodal interfaces where users can visually disambiguate results.
  • Trade-offs are Manageable: The proposed language modeling approach resulted in only a negligible drop in overall ADA performance, indicating that accommodating uncertainty doesn't necessarily come at a high cost.

Key Results

  • Reduced Error Rate: "Something" expressions reduced the relative error rate by 31.8% compared to guessing in the user experiment.
  • Improved Accuracy: "Something" expressions were roughly 2 times more accurate than guesses for Top 1 retrieval and 2.4 times more accurate for Top N retrieval.
  • Minimal Performance Degradation: The language model changes led to a 1% relative reduction in Top 1 accuracy and a 0.4% relative reduction in Top N accuracy on exact listings, deemed negligible.
  • High Recognition Accuracy: The language model recognized "something" with 93% accuracy.

Limitations and Future Work

  • Combining Exact and Approximate Matches: While the two IR methods have complementary strengths, the authors acknowledge that combining them more effectively, especially when users provide incorrect partial knowledge, is an ongoing area of exploration.
  • Data-Driven Generation of Wildcards: Future work includes exploring more data-driven methods for generating "" and "i-" word replacements for training language models, potentially by learning which words users are likely to forget.
  • Enabling Other Uncertainty Expressions: The authors plan to investigate supporting other forms of uncertainty expression, such as alternations (e.g., "steak and grill or grill and steak").
  • User Interface Design: Further usability studies are needed to determine the best way to inform users about the availability and usage of uncertain expressions within the voice search interface.

Summary updated

BibTeX
@inproceedings{paek2008accommodating,
  author = {Tim Paek and Yun-Cheng Ju},
  title = {{Accommodating explicit user expressions of uncertainty in voice search or something like that}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure