TL;DR
This paper introduces a hybrid approach to Chinese definitional question answering (QA) by combining deep linguistic analysis with automatically learned surface patterns. The authors demonstrate that this combination is complementary and significantly improves performance, especially for biographical questions, while also highlighting the challenges of more general definitional questions.
What Problem Does This Paper Address?
Traditional search engines return documents, leaving users to sift through them for specific information. Question Answering (QA) systems aim to directly provide answers. While factoid QA (e.g., "Who is the CEO of Microsoft?") has seen significant progress, definitional QA (e.g., "What is a tsunami?") remains less explored. Definitional QA requires identifying "answer nuggets" that provide a comprehensive understanding of a term, which is more complex than extracting single facts. Existing approaches often rely on either deep linguistic analysis or surface pattern matching, but this paper investigates whether combining these two can overcome their individual limitations for Chinese definitional QA.
Methods
The proposed system uses a hybrid architecture that integrates deep linguistic analysis with automatic pattern learning.
- Deep Linguistic Analysis: The system employs the SERIF linguistic analysis engine to perform tasks like full parsing, name entity detection, relation detection, and co-reference resolution. Key linguistic features extracted include:
- Copula: Linking verbs (e.g., "is") that connect a subject to its definition.
- Apposition: Noun phrases that rename or describe another noun (e.g., "Tony Blair, the British Prime Minister").
- Proposition: Predicate-argument structures that capture relationships between entities.
- Relations: Binary relations between objects (e.g., "spouse-of," "staff-of").
- Surface Pattern Learning: The system automatically learns patterns from annotated data. This involves:
- Annotation: Human annotators mark question terms (QTERM) and answer snippets (ANSWER) in Chinese text.
- Pattern Generalization: Patterns are generalized by replacing sequences with POS tags and applying heuristics to capture common linguistic structures.
- Hybrid Integration: The system combines candidate answers extracted from both deep linguistic analysis and learned patterns. These candidates are then ranked using a predetermined feature ordering, and redundancy is removed.
Datasets
The experiments were conducted on the Chinese TDT4 data (Linguistic Data Consortium, 2002-2003). The authors created question-answer snippets for training and evaluation, separating definitional questions into:
- Biographical (Who-is) questions: Used for pattern learning and testing.
- Other definitional (What-is) questions: Also used for training and testing.
Key Insights and Ideas
- Complementarity is Key: Deep linguistic analysis and automatic pattern learning are not mutually exclusive but rather complementary. Combining them yields better results than either approach alone, especially for biographical QA.
- Pattern Power for Biography: Surface pattern matching is highly effective for answering biographical (Who-is) questions, where answers often follow more predictable linguistic structures.
- The Challenge of "What-is": General definitional (What-is) questions are significantly more challenging. Answers are more diverse and harder to capture with current linguistic features or learned patterns.
- Annotation Efficiency: For biographical QA, a surprisingly small amount of annotation is needed to achieve good performance with learned patterns.
- Useful Linguistic Features: Copulas and appositives are identified as the most useful linguistic features for definitional QA, with relations also playing an important role.
Key Results
- Improved Performance: The hybrid system achieved significant improvements. For "Who-is" questions, combining manual and learned patterns with the baseline (pure linguistic features) resulted in a 18.4% improvement in Rouge score (from 0.3399 to 0.4026). For "What-is" questions, the improvement was smaller (3.5%), reflecting the inherent difficulty of these questions.
- Annotation Efficiency: With only 10% of the training data (549 snippets, ~2 person-days of annotation), learned patterns achieved a Rouge score of 0.3285 on biographical questions, approaching the performance of a fully tuned system.
- Feature Contributions: Learned patterns showed the highest coverage and precision for "Who-is" questions. Copulas and appositives were also strong contributors.
Limitations and Future Work
- "What-is" Question Difficulty: The paper acknowledges that "What-is" questions are much harder due to the diversity and complexity of answers. The authors suggest that finer-grained classification of "What-is" questions (e.g., by entity type like organization, location, substance) could improve performance.
- Limited Pattern Granularity: The current pattern matching relies on simple POS tagging. Future work could explore using automatically derived word clusters to capture richer semantic information.
- Relation Detection Accuracy: The usefulness of relation features could be further enhanced by improving the accuracy of relation detection.
- "What-is" Answer Diversity: The diverse nature of answers for "What-is" questions (e.g., for "international space station," answers include physical attributes, distance, cost) is a significant challenge that requires identifying new, more useful features.
Summary updated