Abstract
This paper proposes a framework for automatically engineering features for two important tasks of question answering: answer sentence selection and answer extraction. We represent question and answer sentence pairs with linguistic structures enriched by semantic information, where the latter is produced by automatic classifiers, e.g., question classifier and Named Entity Recognizer. Tree kernels applied to such structures enable a simple way to generate highly discriminative structural features that combine syntactic and semantic information encoded in the input trees. We conduct experiments on a public benchmark from TREC to compare with previous systems for answer sentence selection and answer extraction. The results show that our models greatly improve on the state of the art, e.g., up to 22% on F1 (relative improvement) for answer extraction, while using no additional resources and no manual feature engineering.
Introduction
Building high-performing Question Answering (QA) systems typically requires two critical steps following document retrieval: answer sentence selection (identifying the correct sentence within a passage) and answer extraction (isolating the specific phrase that answers the question).
Traditionally, these tasks have relied heavily on manual feature engineering or handcrafted rules. For instance, an engineer might define a rule linking the question pattern "What is [Person]'s real name?" to the answer pattern "[Person], whose real name was [Answer]." Such approaches are labor-intensive, fragile, and difficult to scale across different domains. This research introduces a machine learning framework that automatically identifies these structural relationships, eliminating the need for manual intervention.
Methodology
The core of the proposed framework is the application of Partial Tree Kernels (PTK) to enriched linguistic structures. This approach allows the model to "see" and extract discriminative structural fragments (sub-trees) that correlate with correct answers.
Structural Representation
Instead of complex dependency parses, the model utilizes shallow syntactic trees consisting of two levels: word lemmas as leaves and part-of-speech tags organized into chunks (e.g., NP, VP). These trees are then enriched with semantic information:
- Question Classification: Identifying the expected answer category (e.g., Person vs. Location).
- Relational Linking: This "secret sauce" involves adding special tags to link related concepts across the question and answer. If a word appears in both, both nodes receive a
RELtag. If the question asks for a "Person" and the answer contains a "Person" Named Entity, the Question Focus word is linked to that entity using aREL-HUMtag.
By explicitly tagging matching lemmas and semantic types, the framework bridges the lexical gap, allowing the kernel to find patterns across the question-answer pair regardless of surface-level variations.
Experimental Setup
The model was evaluated using the public TREC QA benchmark. Training utilized questions from TREC 8-12, including both a small, manually curated set (TRAIN, 94 questions) and a larger, noisier set (ALL, 1,229 questions). Testing was conducted on TREC 13 questions. The UIUC dataset was used to train the question classifier, while other datasets like SeCo-600 and Mooney GeoQuery supported focus word detection.
Results and Discussion
Answer Sentence Selection
The model significantly outperformed previous state-of-the-art systems. By combining shallow trees with semantic tagging, the system achieved a Mean Average Precision (MAP) of 0.6781 and a Mean Reciprocal Rank (MRR) of 0.7358.
Answer Extraction
The system demonstrated even more substantial gains in answer extraction. When trained on the larger "ALL" dataset, it achieved a massive improvement in F1 score, reaching a 22% relative improvement over previous CRF-based models. This success stems from treating extraction as a ranking task of NP chunks using the kernel-based classifier, rather than a more complex sequence labeling problem.
Conclusion
This work demonstrates that automatic structural learning can be more effective than human intuition for QA tasks. By preserving structural context through tree kernels and bridging semantic gaps with relational tagging, the system outperforms models relying on extensive manual rules or external resources like WordNet. Future work may involve integrating Linked Open Data to further enrich semantic links and moving toward more complex relational reasoning.
Summary updated