Qme! : A Speech-based Question-Answering system on Mobile Devices

Taniya Mishra, Srinivas BangaloreNAACL 2010

TL;DR

This paper introduces "Qme!", a speech-based question-answering system for mobile devices that aims to provide direct answers rather than just links to web pages. It tackles the challenges of mobile input and information access by enabling natural language voice queries and distinguishing between static (timeless) and dynamic (time-sensitive) questions to retrieve answers from archives or live web forms, respectively.

What Problem Does This Paper Address?

Accessing information on mobile devices is a growing trend, but the small screens and keyboards make traditional text-based search and browsing cumbersome and often impractical, especially in hands-busy scenarios. Current systems often return a list of documents, requiring users to sift through them to find the answer. This paper addresses two key issues: 1) the need for a more natural and efficient way to query information on mobile devices, and 2) the desire to provide direct answers to user questions instead of just links. The authors propose a speech-driven system that allows users to ask questions in natural language and receive precise answers, significantly reducing the time and effort required to find information.

Methods

Qme! employs a two-pronged approach to answer questions:

  • Static Questions: For questions with answers that don't change over time (e.g., "What is the speed of light?"), Qme! retrieves answers from a pre-existing archive of human-generated question-answer pairs. This leverages existing knowledge bases for high accuracy.
  • Dynamic Questions: For questions whose answers are time-sensitive (e.g., "What movies are playing near me?"), Qme! classifies them and then queries relevant web forms on the internet to fetch live information. This requires a dynamic/static classifier.

A core innovation is the tight coupling of Automatic Speech Recognition (ASR) and the search component. Instead of using the single "best" output from the ASR, Qme! utilizes a Word Confusion Network (WCN) which represents multiple hypotheses from the ASR. This WCN is then integrated with a Finite-State Transducer (FST) representation of the search index. This co-constraining approach aims to improve the accuracy of both ASR and search by allowing them to inform each other.

The paper also details methods for question retrieval, including TF-IDF and string comparison metrics (Levenshtein, BLEU-score) to match user queries against the question-answer archive. For dynamic/static classification, they explore baseline methods, supervised learning, self-training with bagging, and active learning techniques.

Datasets

The paper uses a large dataset of over a million question-answer pairs harvested from the web for training and evaluation. For evaluation, they created two test sets:

  • Seen Set: 450 QA pairs where queries have an exact match in the database.
  • Unseen Set: 645 QA pairs where queries may not have an exact match, requiring evaluation based on the semantic similarity of retrieved answers to human-annotated "best-matched" answers.

They also used a test set of 250 speech queries for evaluating the tight coupling of ASR and search. For dynamic/static classification experiments, they used a test set of 250 movie-related questions and a large corpus of 500K unlabeled examples.

Key Insights and Ideas

  • Speech as a Natural Mobile Interface: The paper strongly advocates for speech as a more intuitive and practical input method for mobile information access, especially in hands-busy situations.
  • Direct Answers vs. Links: The core value proposition is providing direct answers, which significantly improves user experience and reduces information retrieval time compared to browsing search results.
  • Static vs. Dynamic Question Distinction: Recognizing and handling the difference between static and dynamic questions is crucial for system accuracy and relevance. Static questions benefit from curated archives, while dynamic questions require real-time web querying.
  • Tight ASR-Search Coupling: Integrating ASR hypotheses (via WCNs) with the search index (via FSTs) is a novel approach that demonstrably improves both speech recognition and search accuracy by allowing the components to constrain each other. This moves beyond a simple pipeline.
  • Semi-Supervised Learning for Classification: Techniques like self-training and active learning are effective for building dynamic/static classifiers, especially when labeled data is scarce, by leveraging large amounts of unlabeled data.

Key Results

  • Retrieval Accuracy: For "Seen" queries, retrieval accuracy was very high, with the unigram TF-IDF method achieving 93% in the first position and 100% within the top-10. Other methods also performed excellently.
  • Coherence: For "Seen" queries, n-gram TF-IDF methods and BLEU/Levenshtein metrics showed good coherence for top-5 results, with 4-gram TF-IDF achieving 69.74.
  • Unseen Queries: For "Unseen" queries, over 60% of the "Best-Matched" QA pairs were retrieved in the top-1 position for 405 queries. The WCN-based retrieval method achieved the highest Top-1 accuracy (70.12%) and Top-20 accuracy (78.52%).
  • ASR Improvement: Tightly coupling ASR with the search index improved ASR accuracies by about 1% absolute (e.g., from 77.1% to 78.2% word accuracy on the Dev Set).
  • Dynamic/Static Classification: Active learning achieved the lowest error rate (4.02%) for dynamic/static classification, significantly outperforming baseline and supervised methods. Self-training also showed strong results (8.84%).

Limitations and Future Work

  • QA Archive Size: The authors note that increasing the size of the QA archive is expected to improve coverage for unseen queries.
  • Dynamic Question Answering Coverage: For 240 out of 645 unseen queries, the human expert found no semantically equivalent answer in the repository, indicating limitations in the current database coverage for dynamic questions.
  • GPS Information for Dynamic Questions: While not explicitly detailed as a limitation, the example of "What movies are playing around me?" implies a dependency on user location (GPS), which might not always be available or accurate.
  • Future Work: The paper suggests improving coverage by expanding the QA archive and further refining the dynamic/static classification techniques. The tight coupling of ASR and search is presented as a promising direction for future work.

Summary updated

BibTeX
@inproceedings{mishra2010qme,
  author = {Taniya Mishra and Srinivas Bangalore},
  title = {{Qme! : A Speech-based Question-Answering system on Mobile Devices}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure