TL;DR
This paper presents a modular, data-driven approach to building factoid question answering (QA) systems across multiple languages. They showcase web, mobile, and speech interfaces for their system, demonstrating rapid development and broad accessibility. The core innovation lies in a statistical pattern classification model that efficiently combines a retrieval and a filter model for answer generation.
What Problem Does This Paper Address?
The challenge addressed is the creation of accessible and multilingual factoid question answering (QA) systems. Traditional QA systems often focus on a single language or require extensive manual engineering for each new language. The authors aim to overcome this by developing a system that can be rapidly adapted to new languages using a statistical, data-driven approach, and then made available through various interfaces (web, mobile, speech) to cater to different user needs and contexts.
Methods
The paper's core contribution is a statistical pattern classification approach to QA. The model hypothesizes that an answer $A$ depends on a question $Q$, which is decomposed into two feature sets: $W$ (question-type features like "who," "where") and $X$ (information-bearing features like the core subject of the question). The probability of an answer given a question is modeled as $P(A|Q) = P(A|W, X)$.
To simplify this, they make conditional independence assumptions, leading to an optimization criterion: $\arg \max_A P(A|X) \cdot P(W|A)$. This breaks down into two components:
- Retrieval Model ($P(A|X)$): A language model that estimates the probability of an answer sequence $A$ given the information-bearing features $X$. This models the proximity of the answer to the question's content.
- Filter Model ($P(W|A)$): This model matches answer features with question-type features $W$. It helps re-rank candidate answers by associating question types with valid answer classes (e.g., associating "who" questions with names of people).
The system is built using a modular approach with extensive use of open-source software, allowing for independent improvement and extension to new languages.
Datasets
The paper mentions the following datasets and evaluations:
- TREC 2005 Question Answering Track: Used to evaluate the English QA system.
- NTCIR-3 QAC-1 task: Used to evaluate the Japanese QA system retrospectively.
- Training data: A large corpus of English-language questions was used to train the language models (word and class-based trigram LMs) for the English QA system and the speech interface.
No new benchmarks were introduced; rather, existing evaluation tracks were used.
Key Insights and Ideas
- Rapid Multilingual Development: The data-driven, statistical approach allows for the development of QA systems in five languages in a matter of months, a significant improvement over traditional methods.
- Modularity for Extensibility: The system's modular design, leveraging open-source components, makes it easy to improve individual parts and extend to new languages or interfaces.
- Web Search Integration: Combining QA system output with standard search engine results on the web interface provides users with both direct answers and relevant web pages, enhancing the user experience.
- Speech Interface as a Research Platform: The prototype speech interface, built on VoiceXML and a dedicated speech recognition server, serves as a valuable platform for exploring acoustic, language, and dialog modeling for spoken QA.
- Mobile-First Design Principles: The mobile interface prioritizes speed, minimal data transmission, and concise display, recognizing the constraints of mobile devices.
Key Results
- The web-based QA system is accessible at
http://asked.jpand supports English, Japanese, Chinese, Russian, and Swedish. - The system can be developed for a new language in approximately 10 hours by a proficient developer with access to training data.
- The web interface integrates with a local, open-source web search engine containing about 100 million web pages.
- The average response time for the web interface is around 10 seconds, with web search and QA systems running in parallel.
- The mobile interface displays only the top 20 short answers without pop-up boxes or context to minimize data usage.
- The speech interface uses an adapted Sphinx-4 recognizer and is accessible via U.S. telephone numbers, SIP, Free World Dialup, and SkypeOut.
Limitations and Future Work
- Language Detection: The current web system does not automatically detect the input language; users must manually select the language-specific system.
- Mobile Interface Enhancement: While the mobile interface removes entire pages to save data, the authors aim to allow users to click on an answer and be taken directly to the relevant part of the page, rather than loading the whole page.
- Speech Interface Performance: The goal is to match the QA performance of the speech interface to that of the web and mobile interfaces. This will be achieved through acoustic, language, and dialog model adaptation on the speech side, and by making the QA system more robust to underspecified or erroneous questions.
- Unsupervised System Adaptation: The authors hope to use user click data on the web interface to enable unsupervised system adaptation in the future.
Summary updated