Abstract
Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains. This paper introduces and studies the notion of essential question terms with the goal of improving such QA solvers. We illustrate the importance of essential question terms by showing that humans’ ability to answer questions drops significantly when essential terms are eliminated from questions. We then develop a classifier that reliably (90% mean average precision) identifies and ranks essential terms in questions. Finally, we use the classifier to demonstrate that the notion of question term essentiality allows state-of-the-art QA solver for elementary-level science questions to make better and more informed decisions, improving performance by up to 5%. We also introduce a new dataset of over 2,200 crowd-sourced essential terms annotated science questions.
Introduction
Current question-answering (QA) systems are often "distracted" by the wordy, redundant, or irrelevant language common in complex questions. For example, in a standardized science test question like "One way animals usually respond to a sudden drop in temperature is by...", a human immediately identifies "drop in temperature" and "respond" as the core components. However, many QA systems struggle to distinguish these from "fluff" like "usually" or "one way." An Information Retrieval (IR) system might prioritize "animals usually respond," while a logic-based system might focus on "animals" and ignore the temperature context entirely.
This paper addresses this gap by introducing a mechanism to identify essential question terms—the core words required to find an answer. This is particularly vital in domains like elementary science, where questions are often multi-sentence and highly context-dependent.
Methodology
The researchers framed "essentiality" as a supervised learning problem.
Defining Essentiality
Three criteria were established for human annotation:
- Altering an essential term must change the question's core meaning.
- Dropping non-essential terms should not change the correct answer.
- Grammatical correctness is secondary to preserving meaning.
The ET Classifier
A linear Support Vector Machine (SVM) classifier was developed using 120 features across three categories:
- Syntactic: Dependency parses and Part-of-Speech (POS) tags.
- Semantic: Brown clusters and lists of scientific terms.
- Statistical: Pointwise Mutual Information (PMI) measuring co-occurrence with potential answers.
Solver Integration
The classifier was integrated into two solvers: an IR Solver, where search queries were restricted to terms deemed essential, and TableILP (Inference Solver), which used a "cascading" approach to progressively relax word requirements if an answer was not found using only essential terms.
Key Insights and Validation
The authors validated their labels by demonstrating that removing just 12% of "essential" terms causes human accuracy to plummet. In contrast, humans can still answer 65% of questions even after 80% of "non-essential" terms are removed.
Other findings include:
- POS Tags are Deceptive: Unlike many NLP tasks, proper nouns (NNP) are not always essential in science QA; fewer than half were marked as such.
- Science Terms Rule: Words appearing in a pre-defined list of scientific terms (e.g., "gravity") had a 76.6% essentiality rate.
- Context is King: Essentiality is dynamic; the classifier learned that a word's importance depends on the surrounding sentence and the scientific concept being tested.
Results
The ET Classifier achieved a 90% Mean Average Precision (MAP) in ranking essential terms, significantly outperforming unsupervised baselines like PMI or simple frequency counts.
The classifier remained robust on "unseen" terms, maintaining an F1 score of 0.78. Integration into solvers yielded significant gains: the IR solver saw a 5% performance increase on noisy datasets, and the cascading approach for TableILP corrected 41.7% of previous mistakes on a curated set of answerable questions.
Conclusion
While "essentiality" remains somewhat subjective (Fleiss’ kappa of 0.58), this research demonstrates its utility in improving QA robustness. Future work may explore how these features translate to other specialized domains like medicine or law, and investigate end-to-end differentiable methods for integrating essentiality into neural QA architectures.
Summary updated