TL;DR
This paper introduces "Qme!", a speech-driven question-answering system designed for mobile devices that bridges the gap between the easily searchable "surface web" and the vast, unindexed "Deep Web." It tackles the challenge of accessing dynamic, form-based information on mobile devices by using spoken language to query these hidden repositories.
What Problem Does This Paper Address?
The internet is a treasure trove of information, but a significant portion of it, known as the Deep Web, remains inaccessible to traditional search engines. This Deep Web consists of proprietary databases that are typically accessed via web forms, often containing dynamic and frequently updated information (e.g., flight schedules, movie times, restaurant reservations). Accessing this information on mobile devices is particularly challenging due to small screens and limited input capabilities, making typing tedious and often impractical. Existing systems primarily focus on the surface web, leaving users without an efficient way to query the Deep Web using natural, spoken language on their mobile devices.
Methods
The Qme! system employs a multi-stage approach:
- Speech Recognition: It first converts spoken queries into text using an Automatic Speech Recognition (ASR) system, which is adapted for mobile device speech and uses a four-gram language model.
- Query Classification: The system then classifies queries into two categories:
- Static Questions: These have answers that don't change over time and are retrieved from a pre-existing archive of question-answer pairs.
- Dynamic Questions: These require accessing external, often Deep Web, sources.
- Dynamic Question Handling: For dynamic questions, Qme! uses a two-stage machine learning classifier:
- Topic Classifier: Identifies the broad topic of the question (e.g., movies, sports).
-
Dynamic/Static Classifier: Further categorizes the question within its topic. Once classified as dynamic, the system parses the query to extract relevant keywords and parameters. It then uses these parameters to query specific web forms on trusted aggregator websites (e.g., Fandango for movie information). The retrieved HTML results are then parsed and reformatted for optimal display on mobile devices.
-
Static Question Handling: For static questions, the system leverages a custom search engine built using finite-state transducers (FSTs). This FST-based index represents question-answer pairs, allowing for efficient retrieval even from weighted word lattices produced by the ASR.
Datasets
- QA Archive: Used for retrieving answers to static questions.
- Training Data for Topic Classifier: Over one million questions manually labeled with topic information.
- Test Data for Topic Classifier: 15,000 randomly selected questions.
- Test Corpus for Dynamic/Static Classification: A 250-question test set drawn from the "Movies" topic.
- Data for Dynamic Question Answering: The paper mentions using data from ChaCha for the system.
Key Insights and Ideas
- Unified Interface for Surface and Deep Web: The core innovation is providing a single, natural interface (spoken language) to access both easily indexed and hidden web content on mobile devices.
- Importance of Dynamic Question Classification: Accurately distinguishing between static and dynamic questions is crucial for routing queries to the appropriate retrieval mechanism. The paper highlights that simple keyword-based classification (like temporal indexicals) is insufficient, necessitating more sophisticated machine learning approaches.
- Leveraging Mobile Device Capabilities: The system effectively utilizes mobile device features like GPS for inferring location when not explicitly provided in a query.
- FSTs for Efficient Static Search: Using finite-state transducers for indexing static questions offers an efficient alternative to traditional text search engines, especially when dealing with ASR output lattices.
- Deep Web Data Reformatting: Recognizing that raw HTML from Deep Web sources is not mobile-friendly, the system's ability to parse and reformat results is a practical necessity.
Key Results
- Topic Classification: The MaxEnt classifier achieved significantly lower error rates than a baseline model, with an error rate of 12.23% when considering the top two predicted topics.
- Dynamic/Static Classification: The Active-learning approach yielded the lowest error rate (4.02%) for dynamic/static classification on the movie-related test set, outperforming baseline and self-training methods.
- Regular-expression Recognizer: Initially identified only 3.5% of a dataset as dynamic, while human annotation suggested an average of 13.5%, underscoring the limitations of simple rule-based methods.
Limitations and Future Work
- Limited Vertical Searches: Currently, the system only incorporates vertical searches for three broad topics: Movies, Mass Transit, and Yellow Pages. Future work would involve expanding this to cover more topics.
- Web Form Parsing Complexity: The paper acknowledges that parsing different web forms and extracting parameters can be complex, especially when query parameters are not explicitly stated. Further details on this extraction are presented in other work by the authors.
- SOAP Protocol: While the system currently parses HTML, the authors note that not all aggregator sites use SOAP (Simple Object Access Protocol), which could be a more standardized approach for data retrieval.
- ASR Model Adaptation: The ASR model used is adapted to speech collected from mobile devices, but further improvements in robustness to noise and varying acoustic conditions could be beneficial.
Summary updated