TL;DR
This paper introduces DEFQA, a novel unsupervised learning method for extracting single-sentence definitions from web pages in response to definition-seeking questions. By leveraging online encyclopedias and dictionaries to automatically generate training data, DEFQA significantly outperforms previous methods and improves the ability of search engines to handle this common type of query.
What Problem Does This Paper Address?
Question Answering (QA) systems aim to provide direct answers to user queries, often by extracting snippets of text from documents. While effective for factual questions (e.g., "Who is the president?"), QA systems struggle with "definition" questions (e.g., "What is gasohol?"). These questions often lack specific entity types and contain only a single term, making them difficult to process with traditional methods. Existing approaches often rely on manually labeled data, which is expensive and time-consuming to create, especially for new document collections. This paper addresses the challenge of building effective definition-answering systems that can learn from readily available web data without extensive manual annotation.
Methods
The core of the proposed method, DEFQA, is a Support Vector Machine (SVM) trained to classify text snippets (windows) as either definitions or non-definitions. The key innovation lies in its unsupervised learning approach for generating training data.
Instead of relying on human annotators, DEFQA uses online encyclopedias and dictionaries to create a large corpus of "encyclopedia definitions" for a given target term. It then compares candidate snippets from web pages to these encyclopedia definitions using a similarity measure.
- Similarity Measure: This measure quantifies how closely a candidate snippet's wording matches the wording of encyclopedia definitions. It considers word frequency within the definitions and inverse document frequency in a large corpus (British National Corpus) to identify words that are both common in definitions and rare in general text.
- Automatic Tagging: Snippets with high similarity to encyclopedia definitions are tagged as positive examples (definitions), while those with low similarity are tagged as negative examples (non-definitions). Snippets with intermediate similarity are excluded to ensure high confidence in the training data.
- Feature Engineering: The SVM uses a vector representation of each window, incorporating attributes such as:
- SN (Snippet Number): The position of the snippet within the document.
- WC (Word Centroid): How well the snippet's words match the most frequent words across all candidate definitions.
- RK (Rank): The ranking of the web page by the search engine.
- Manual Patterns: Predefined lexical patterns (e.g., "target, a/an/the").
- Automatic Patterns: Lexical patterns automatically extracted from training data, capturing common definition structures on the web.
Datasets
- Training Data Generation: The method leverages online encyclopedias and dictionaries (e.g., Wikipedia, Google's 'define:' feature) to generate "encyclopedia definitions."
- Training Windows: Windows are extracted from web pages returned by a search engine (Altavista was used in experiments) for a set of training target terms.
- Evaluation Data: The evaluation was performed on 81 unseen target terms, deliberately chosen from an online encyclopedia to aid human assessors in judging snippet quality.
Key Insights and Ideas
- Unsupervised Learning for Definitions: The most significant insight is the successful application of an unsupervised learning paradigm to the challenging task of identifying definitions on the web. This bypasses the need for costly manual annotation.
- Leveraging External Knowledge: Using online encyclopedias and dictionaries as a "ground truth" for generating training data is a clever way to bridge the gap between structured knowledge and unstructured web text.
- Web-Specific Patterns: The automatic acquisition of lexical patterns from web pages, rather than solely relying on patterns from structured sources like TREC documents, allows DEFQA to better capture the nuances of definitions found in the wild.
- Single-Snippet Focus: The paper argues for the continued relevance and practical utility of single-snippet definitions, highlighting easier inter-assessor agreement compared to multi-snippet evaluations.
Key Results
- Viability of Unsupervised Method: The proposed unsupervised method is shown to be viable and effective.
- Outperforms TREC-based Training: Training DEFQA on automatically generated web data (DEFQA) significantly outperformed training it on TREC questions and documents (DEFQAt).
- Improved Search Engine Performance: DEFQA helps search engines handle definition questions significantly better than without it.
- Quantitative Performance: In evaluation with a strict k=1 (one snippet per question), DEFQA answered correctly approximately 6 out of 10 definition questions. This is considered remarkable given the difficulty of the task and the single-snippet constraint.
- Baseline Comparison: Both DEFQA and DEFQAt significantly outperformed baseline methods that simply returned the first or a random snippet from a web page.
Limitations and Future Work
- Positive Precision Noise: The similarity method, while effective, can introduce some noise in positive training examples (i.e., snippets tagged as definitions that are not). The authors acknowledge this and suggest potential improvements.
- Threshold Selection: The choice of similarity thresholds (t+ and t-) is crucial for balancing precision and recall in training data generation.
- Future Work:
- Improve positive precision of training examples, potentially by combining the similarity method with centroid-based approaches (like Cui et al.).
- Investigate the effect of including more automatically acquired patterns.
- Study the impact of increasing the number of training target terms (q).
- Incorporate attributes related to the layout and authority of web pages.
- Extend the system to generate multi-snippet definitions.
Summary updated