Examining the Role of Statistical and Linguistic Knowledge Sources in a General-Knowledge Question-Answering System

Claire Cardie, Vincent Ng, David Pierce, Chris BuckleyNAACL 2000 · DOI

TL;DR

This paper explores how to build a question-answering (QA) system that can find short answers to general knowledge questions within a large text collection. They found that while standard information retrieval (IR) is good at finding relevant documents, even simple linguistic filters significantly boost performance, especially when integrated with IR.

What Problem Does This Paper Address?

The ultimate goal of Question Answering (QA) is to build systems that can answer any question posed in natural language. However, current text processing systems struggle with this broad scope. This paper focuses on a more restricted, yet still valuable, problem: given a large collection of text and English questions, find the answer within that collection. The specific constraints are that the answer must exist in the collection, all supporting information for the answer must be in a single document, and the answer itself must be short (less than 50 bytes). This last constraint effectively biases answers towards noun phrases. The authors aim to understand the relative contributions of statistical (IR-based) and linguistic (NLP-based) knowledge sources in such a system.

Methods

The system architecture is modular, consisting of two main components: IR subsystems and linguistic filters.

  • IR Subsystems:
    • Document Retrieval: Uses the Smart vector space model to represent both questions and documents as vectors. Cosine similarity is used to rank documents by relevance to the question.
  • Query-Dependent Text Summarization: Documents are divided into chunks (sentences or paragraphs), and their vector representations are compared to the question's vector. The most similar chunks are selected to form a summary, aiming to focus subsequent processing on relevant passages. This also uses vector space methods.
  • Linguistic Filters: These are applied to the summarized extracts to refine answer hypotheses.
  • Noun Phrase (NP) Filter: Identifies noun phrases within the summary extracts as potential answers. This is a simple filter based on a learned noun phrase grammar.
  • Semantic Type Checking Filter: Determines the semantic type of the answer requested by the question (e.g., person, place, time) and checks if candidate noun phrases are semantically compatible using WordNet or heuristics.
  • Syntactic Preferences for Ordering Summary Extracts: A novel approach that reorders summary extracts based on a combined score that considers both vector space similarity and the presence of linguistic relationships (initially, just base noun phrases) between the question and the extract.

Datasets

  • TREC8 Development Corpus: Used for tuning system parameters and initial experiments (38 questions).
  • TREC8 Test Corpus: Used for final evaluation (200 questions).
  • Indexed Collection: TREC disks 4 and 5 (excluding Congressional Records).

Key Insights and Ideas

  • IR is a strong starting point: Standard IR techniques (like vector space models) are surprisingly effective at quickly and accurately locating relevant documents for general knowledge questions.
  • Summarization's double-edged sword: Query-dependent text summarization provides only small performance gains and can severely limit recall if inaccurate, as answers might fall outside the summarized passages. However, it's crucial for allowing linguistic filters to focus.
  • Weak linguistic knowledge is powerful: Even very simple linguistic filters (like identifying noun phrases or checking semantic types) can lead to substantial improvements over purely IR-based methods, especially when integrated well.
  • Integration is key: The effectiveness of linguistic filters is highly dependent on how well they are integrated with the statistical preferences provided by the IR subsystems.
  • Chunking matters: The way answers are presented (e.g., 50-byte chunks vs. single phrases) significantly impacts performance, suggesting that while filters identify good candidates, further processing is needed to promote the best ones.

Key Results

The paper presents a series of evaluations showing performance improvements as more components are added:

  • Baseline (Smart Vector Space Model): 3/38 (7.9%) correct on the development set, 29/200 (14.5%) on the test set.
  • With Query-Dependent Text Summarization: Performance increases slightly to 4/38 (10.5%) and 45/200 (22.5%).
  • With NP Filter: Performance improves to 7/38 (18.4%) and 50/200 (25.0%).
  • With Semantic Type Filter: A dramatic jump to 21/38 (55.3%) and 86/200 (43.0%). This is the most significant gain from a single linguistic filter.
  • Final System (Syntactic Ordering + NPs + Semantic Type): Achieves 22/38 (57.9%) and 91/200 (45.5%) correct.

Notably, the semantic type filter's heuristics, designed on the development corpus, did not generalize as well to the test corpus, explaining the smaller relative gain there.

Limitations and Future Work

  • Generalization of Heuristics: The semantic type checking filter's heuristics need improvement to generalize better to different question sets.
  • Named Entity Recognition: The system lacks a robust named entity tagger, which is identified as a critical area for future work.
  • Coreference Resolution and Relevance Feedback: Incorporating automatic relevance feedback and coreference resolution could improve retrieval and summarization.
  • Paragraph Summaries: Paragraph-based summaries showed better coverage on the test corpus and might be more suitable for sophisticated linguistic filters.
  • Phrase Relation Confirmation: The current system doesn't confirm phrase relationships from the query, which is seen as a limitation that severely restricts performance. Future work will focus on augmenting named entity identification with WordNet and confirming phrase relations.

Summary updated

BibTeX
@inproceedings{cardie2000examining,
  author = {Claire Cardie and Vincent Ng and David Pierce and Chris Buckley},
  title = {{Examining the Role of Statistical and Linguistic Knowledge Sources in a General-Knowledge Question-Answering System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2000},
  doi = {10.3115/974147.974172},
}
Figure