TL;DR
This paper tackles the challenge of automatically classifying questions in Question Answering (QA) systems. The authors propose using Support Vector Machines (SVMs) with error-correcting output codes (ECOC) to handle the multi-class nature of question types, achieving competitive performance with significantly less linguistic expertise than prior methods.
What Problem Does This Paper Address?
Question Answering (QA) systems aim to provide direct answers to user queries. A crucial first step in many QA systems is classifying the type of question being asked. This classification helps in formulating the correct query to search for an answer and in extracting the answer itself. Traditionally, this classification relied heavily on domain experts crafting intricate sets of regular expressions and keywords. This approach is time-consuming, labor-intensive, and difficult to adapt to new domains. The authors sought to replace this manual effort with a machine learning approach that learns classification patterns directly from labeled data, making the process more efficient and portable.
Methods
The core of the paper's approach lies in using Support Vector Machines (SVMs) for question classification. Since SVMs are inherently binary classifiers, the authors employ Error Correcting Output Codes (ECOC) to extend them to a multi-class problem. This involves assigning a unique binary codeword to each question type. The system then trains multiple binary SVMs, one for each bit in the codewords.
The process involves:
- Preprocessing: Cleaning the text by keeping only alphabetic characters, converting to lowercase, removing common "stop words," and stemming words.
- Indexing: Representing questions as frequency-weighted vectors of terms (words).
- Dimensionality Reduction: Using Singular Value Decomposition (SVD) to reduce the high dimensionality of the term space, aiming to mitigate noise and overfitting.
- Classification: Training multiple binary SVMs using ECOC. The final classification is determined by combining the outputs of these SVMs.
The authors also explore the use of Named Entity (NE) tags as additional features, augmenting the basic word-based representation.
Datasets
- Training Data: 5500 labeled questions, made publicly available.
- Test Data: 500 labeled questions from TREC-10.
- Question/Answer (Q/A) Type Taxonomy: The authors developed a taxonomy of 50 question types, which is illustrated in Figure 1. This taxonomy is hierarchical and can be adapted based on specificity and coverage requirements.
Key Insights and Ideas
- SVMs + ECOC for Multi-class QC: The combination of SVMs with ECOC provides a robust framework for multi-class question classification, effectively handling the problem without requiring complex, custom-built classifiers for each class.
- Reduced Reliance on Linguistic Expertise: The paper demonstrates that a high level of performance can be achieved with minimal linguistic analysis, contrasting with prior methods that relied on computationally expensive syntactic and semantic processing.
- Feature Representation Matters: While NE tags improved performance, the authors found that SVD transformation of the term space, intended to create a more optimal basis, surprisingly led to a performance loss compared to using the raw term space as the basis. This suggests that the original term space, despite its high dimensionality, might retain more discriminative information for this specific task.
- "Words Only" Performance: The "words only" performance of their system is competitive, highlighting the effectiveness of the chosen features and classification method even without advanced linguistic features.
Key Results
- The system achieved a classification accuracy of 82.0% using 1-gram features and no NE tags, and 81.4% with 7 NE tags.
- When using SVD transformation, the performance slightly decreased, with the best result being 79.8% using SVD, 1000 dimensions, and NE-7 tags.
- The authors' "words only" approach achieved a performance comparable to prior work that used more complex features, with their "words only" performance at 79.4% (for 1-gram) compared to a reported 52.4% for a similar "words only" baseline in related work.
- The best result reported in related work (Li and Roth, 2002) for a flat classifier was 84.0%, using a comprehensive set of features including words, NEs, POS tags, chunks, heads, and related words. The authors' approach achieves comparable performance with significantly less feature engineering.
Limitations and Future Work
- SVD Performance Loss: A key limitation observed is the performance degradation when using SVD for dimensionality reduction. The authors note that the basis defined by SVD might not be optimal for classification, and this warrants further investigation.
- NE Tagging Performance: The performance of the 29-NE tagger was worse than the 7-NE tagger, and some discriminative words were collapsed into broad tags, impacting classification.
- Future Work: The authors suggest that further exploration of dimensionality reduction techniques and feature combinations could lead to even better performance. They also highlight the potential for their approach to encourage wider adoption of readily available free software and question corpora for QC.
Summary updated