TL;DR
This paper explores how to combine the outputs of multiple question-answering (QA) systems to achieve better performance than any single system. They investigate various distance-based methods and metrics for combining answers, demonstrating that combining diverse system outputs can indeed improve accuracy, though not yet surpassing the best individual TREC system.
What Problem Does This Paper Address?
The field of question answering (QA) aims to build systems that can automatically find answers to questions posed in natural language. While individual QA systems have been improving, this paper asks a fundamental question: can we achieve even better performance by combining the outputs of multiple, diverse QA systems? This is analogous to asking how well a "community" of QA systems would perform. Furthermore, establishing lower bounds on achievable performance is crucial for understanding the inherent difficulty of the task and setting realistic goals for future research. The authors draw inspiration from the ROVER system in speech recognition, which successfully combined multiple recognizer outputs.
Methods
The core idea is to treat the output of multiple QA systems as a collection of potential answers to a given question and then select the "best" answer from this collection. The authors focus on distance-based combining methods.
- Input: Each answer consists of the answer string and the identifier of the document it came from.
- Combining Techniques:
- Voting: A simple method where the most frequently suggested answer wins.
- Centroid Method: A more sophisticated approach inspired by Gusfield (1993), which finds a "centroid" answer that minimizes the sum of distances to all proposed answers. This method is appealing because it can be bounded and reduces to simple voting with an "exact match" distance.
- Distance Metrics: To measure similarity between answers, they explored:
- Levenshtein edit distance: A standard metric for string similarity.
- Tanimoto distance on sets: Measures the similarity between two sets based on their intersection and union.
- Generalized Tanimoto distance for multisets: An extension to handle repeated elements.
- N-gram features: Both word and character n-grams were used as atomic units for similarity calculations.
- Priors: To account for system reliability and document source, they incorporated "source priors." These priors represent the expected accuracy of systems when generating answers from different document sources (e.g., Associated Press, New York Times, NIL). These priors were used to scale distance measures.
Datasets
- TREC 2002 Question Answering Task Data: The experiments were conducted using data from the TREC 2002 QA task.
- Full Dataset: 67 different QA systems or variants participated, generating approximately 67 * 500 = 33,500 answers.
- Development Set: A randomly selected subset of 100 questions (approximately 6700 proposed answers) was used for developing and tuning the combining techniques.
- Test Set: The remaining 400 questions were used for final evaluation.
Key Insights and Ideas
- Diversity is Key: Combining diverse system outputs is a promising avenue for improving QA performance, as different systems are likely to make different types of errors.
- Distance Matters: The choice of distance metric significantly impacts the effectiveness of combining techniques. N-gram-based features and generalized Tanimoto distance proved effective.
- Priors Improve Performance: Incorporating prior knowledge about system reliability (e.g., based on document source) is crucial for scaling distance measures and improving accuracy.
- The "NIL" Challenge: A significant portion of answers were "NIL" (indicating the question was unanswerable), highlighting the conservative nature of some systems and the need to handle these cases.
- Dev Set Size: The relatively small size of the development set (100 questions) might limit the generalizability of findings, although rank orderings were consistent between dev and test sets.
Key Results
- The best-performing combination system used character strings of length 5 and less as features, a multiset Tanimoto distance, scaled with document source priors.
- This best selector achieved 76% strict accuracy and 90% average precision on the test set (400 questions).
- A simple "exact string match" voting scheme achieved 42% strict accuracy and 65% average precision on the test set.
- While the best combined system did not surpass the best individual TREC system (which achieved 86% strict accuracy and 90% average precision on the test set), the results demonstrate the effectiveness of the combining approach.
- The "Loose" evaluation metric (which relaxed strict correctness criteria) showed higher performance for all methods, indicating the difficulty of achieving perfectly exact and justified answers.
Limitations and Future Work
- Limited Evidence Sources: The authors acknowledge not having access to several potentially useful evidence sources:
- System Confidence Scores: Submissions often included implicit confidence estimates, which could be leveraged by weighting answers.
- System IDs: Knowing which system produced which answer would allow for system-level priors, leveraging the known reliability of certain systems.
- Question Features: Future work could explore incorporating question features to learn which systems are better suited for specific question types.
- Dev Set Size: The authors suggest that the dev set size might be too small for drawing definitive conclusions on the relative quality of techniques.
- Exploiting More Diversity: Further research could explore more sophisticated ways to exploit the diversity of system outputs and evidence sources.
Summary updated