Automatic Question Answering: Beyond the Factoid

Radu Soricut, Eric BrillNAACL 2004

TL;DR

This paper introduces a question answering (QA) system that moves beyond simple factoid questions to handle more complex, FAQ-like queries. It achieves this by employing a noisy-channel architecture trained on a massive corpus of web-scraped question-answer pairs, demonstrating improved performance on non-factoid questions.

What Problem Does This Paper Address?

Traditional Question Answering (QA) systems have largely focused on "factoid" questions – those with short, definitive answers (e.g., "What is the capital of France?"). While significant progress has been made in this area, a large portion of real-world questions people ask are more complex and require longer, more nuanced answers, often found in FAQ sections. These "non-factoid" questions, such as "How does a film qualify for an Academy Award?", are harder to answer with existing methods because they lack the simple, direct answer format. This paper aims to bridge this gap by developing a QA system capable of handling these more general and complex questions, which have greater practical and economic potential.

Methods

The core of the proposed QA system is a noisy-channel architecture. This approach models the process of generating a question from an answer. The system comprises four main modules:

  • Question2Query: This module transforms natural language questions into phrase-based queries suitable for search engines. It uses a statistical chunker trained on answer-side statistics to bridge the lexical gap between questions and answers, aiming to create more effective search queries.
  • SearchEngine: This module interfaces with existing search engines (like MSNSearch and Google) to retrieve relevant documents based on the generated query.
  • Filter: This module processes the search results, reducing the volume of documents to a manageable set and segmenting the text into sentences. A crucial aspect here is a filtering step to prevent the system from returning answers that are too similar to the reference answers in the evaluation set, thus avoiding artificially inflated scores.
  • AnswerExtraction: This is the most critical module for handling non-factoid questions. It employs two main algorithms:
  • N-gram Co-Occurrence Statistics (NG-AE): This method uses the BLEU score to measure the overlap between question terms and potential answers, aiming to find the best matching answer.
  • Statistical Translation (M1e-AE): This algorithm attempts to bridge the "lexical chasm" between question and answer terms using a noisy-channel model inspired by statistical machine translation. It models the probability of a question given an answer, leveraging both an answer generation model and an answer-to-question translation model. They explore variations like Model 0 and Model 1, with Model 1 being more sophisticated but initially performing worse due to distribution issues.

Datasets

  • Training Data: The authors constructed a large training corpus of approximately 1 million question/answer pairs by scraping FAQ pages from the Web. This is significantly larger than previous datasets used for similar tasks.
  • Evaluation Data: A set of 115 test questions was used for evaluation. These questions were specifically chosen to be non-factoid and covered a wide variety of topics.

Key Insights and Ideas

  • The Lexical Chasm is Real and Problematic: The paper strongly emphasizes that the difference in vocabulary and phrasing between questions and answers is a major hurdle, especially for complex questions. Simply reformulating questions often doesn't help and can even hurt performance.
  • Statistical Translation is Key for Non-Factoids: Bridging the lexical gap using techniques inspired by machine translation (like the noisy-channel model) is crucial for answering non-factoid questions effectively.
  • Large-Scale Data is Essential: The success of their approach hinges on the massive training corpus of 1 million FAQ pairs, highlighting the importance of data volume for learning robust models.
  • Chunking Questions Improves Search: Using a statistical chunker to create phrase-based queries for search engines significantly improves the retrieval of relevant documents compared to using the raw question.
  • Answer Quality vs. Answer Type: The evaluation revealed that sometimes the system provided answers that were factually correct but of the wrong "type" (e.g., a "how-to" answer for a "what-is" question), indicating a need for better question type understanding.

Key Results

  • The QA system achieved a performance score of 0.38 using the M1e-AE (Model 1, segmented question) algorithm on the test set.
  • Using the statistical chunker in the Question2Query module improved performance by 0.05 (from 0.18 to 0.23 for NG-AE, and 0.34 to 0.38 for M1e-AE).
  • The Filter module's evaluation showed that considering the top 10 hits from the search engine provided a significant performance ceiling increase (from 0.36 to 0.46), with diminishing returns for considering more hits (up to 50).
  • The AnswerExtraction module using the M1e-AE (Model 1) algorithm, trained on the large corpus, outperformed simpler models and achieved a performance ceiling of 0.46 when using an oracle-informed approach.

Limitations and Future Work

  • Performance Ceiling: The system's performance is limited by the quality and relevance of documents returned by the search engine. The "oracle" evaluation showed a ceiling of around 0.46, meaning even perfect answer extraction would not yield higher scores.
  • Question Type Understanding: The system sometimes struggles with identifying the correct "type" of answer required for a question (e.g., distinguishing between "how-to" and "what-is" questions).
  • Interactions Between Modules: Many errors arise from the complex interactions between modules, rather than a single module's failure.
  • Future Work: The authors suggest developing a question typology for the FAQ domain as a crucial next step, similar to how factoid QA systems benefit from question type classification. They also propose further fine-tuning QA architectures based on the lessons learned.

Summary updated

BibTeX
@inproceedings{soricut2004automatic,
  author = {Radu Soricut and Eric Brill},
  title = {{Automatic Question Answering: Beyond the Factoid}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure