TL;DR
This paper presents a multimodal interface for spoken question-answering (QA) on mobile devices, combining a customized dictation engine, a predictive text correction interface, and the AnswerBus QA system. They demonstrate that spoken QA is feasible, even with recognition errors, and propose modifications to improve robustness.
What Problem Does This Paper Address?
The proliferation of mobile devices with small screens presents a challenge for traditional web browsing and document retrieval. While question-answering systems offer a more direct way to find information, inputting complex queries via touch interfaces can be slow and cumbersome. This paper tackles the problem of enabling efficient and accurate question input for open-domain QA systems on these devices, specifically by exploring the use of speech as an input modality. The authors highlight that while spoken document retrieval has seen progress, applying speech to natural language question answering, which requires more precise input, has not been extensively explored.
Methods
The authors developed a multimodal interface comprising three core components:
- Speech Recognizer: They utilized Dragon NaturallySpeaking 6.1, a commercial dictation engine. Crucially, they customized its language models by training it on a large corpus of questions (280,000 from Schofield, 2003). This adaptation aims to improve recognition accuracy for the specific linguistic structures found in questions.
- Predictive Typing Interface: To address speech recognition errors, they implemented a Web-based predictive typing interface. This interface uses a 3-gram language model (trained on the same question corpus) to suggest corrections as the user types. It's designed as a thin JavaScript client that intercepts keystrokes and uses CGI requests to fetch predictions, which appear as clickable hyperlinks to modify the question. This allows for quick correction of mistranscriptions.
- Question-Answering System (AnswerBus): The core QA engine is AnswerBus, a publicly available Web system. The authors made two key modifications to adapt it for spoken input:
- Speed Optimization: They implemented a multi-level caching system (recently asked questions and semi-structured web documents) to reduce response times, which is critical for spoken interfaces and mobile devices where latency is higher.
- Output Transformation: The original engine provided sentence-based hyperlinks. For spoken output, they transformed this to offer plain text as an alternative to HTML.
Datasets
- Training Data:
- A corpus of 280,000 questions (described in Schofield, 2003) was used to customize the language models for the Dragon NaturallySpeaking speech recognizer and to train the 3-gram language model for the predictive typing interface.
- Evaluation Data:
- 200 test questions from the TREC 2002 QA track (Voorhees, 2002) were used. The system's accuracy was evaluated by comparing its returned snippets with the reference answers.
Key Insights and Ideas
- Speech is Viable for QA, but Errors Matter: The study demonstrates that spoken questions can be used for automatic QA, but the precision of the returned answers is significantly impacted by speech recognition errors. This underscores the need for robust error correction mechanisms.
- Linguistic Structure of Questions is Exploitable: The authors' prior work suggests questions have a limited lexical structure that can be leveraged for better speech recognition. Their customization of language models for questions supports this hypothesis.
- Predictive Correction is Crucial: The predictive typing interface is highlighted as a key component for making spoken QA practical. It allows users to quickly correct mistranscriptions, thereby improving the accuracy of the final query sent to the QA system.
- QA Systems Need Adaptation for Speech: Standard QA systems, like AnswerBus, are not inherently designed for spoken input. Modifications such as optimizing response speed and adapting output formats are necessary for a good spoken interface experience.
- Homograph Confusion is a Recognizable Problem: The paper points out specific types of homograph confusion (e.g., plural vs. possessive nouns, numbers vs. numerals) that are common and can be addressed by QA systems to improve robustness.
Key Results
- Accuracy with Misrecognized Speech: When tested with misrecognized speech, the system achieved 39% and 26% accuracy for two speakers, respectively. This is significantly lower than the 58% and 60% accuracy achieved with verbatim typed input.
- Impact of Errors: Table 3 shows that for Speaker 1, 12 answers were degraded due to input errors, while for Speaker 2, 34 answers were degraded. Only 5 answers were improved by the "dodgy input" (implying that the errors sometimes coincidentally led to better results, though this was rare).
- Potential for Improvement: The authors suggest that features like automatic spelling correction and morphological stemming could further improve robustness to speech recognition errors.
Limitations and Future Work
- No Automatic Error Checking: The current system performs no automatic checking or correction of spelling and no morphological stemming. The authors believe these features would improve robustness.
- Homograph Handling: While they identified common homograph confusions, their system did not explicitly compensate for them in the evaluation.
- Query Expansion: More extensive query expansion using synonyms or other orthographic forms is identified as a potential area for improvement, though it is noted as being trickier to implement.
- Integration and Porting: The authors plan to continue tightening the integration of system components and porting the interface to phones and Palm or Pocket PC devices.
Summary updated