TL;DR
This paper introduces a two-stage framework for multiple-choice question answering (QA) that combines traditional information retrieval (IR) with deep learning inference models. By transforming QA into a natural language inference task and using an ensemble of models pre-trained on diverse datasets, the authors achieved a significant performance boost on complex science questions that require reasoning rather than just keyword matching.
What Problem Does This Paper Address?
While modern AI assistants like Alexa or Google Assistant are excellent at answering "factoid" questions (e.g., "What is the capital of France?"), they struggle with complex, non-factoid questions found in middle-school science exams. These questions often require inference—the ability to connect disparate pieces of information to reach a logical conclusion.
Standard retrieval-based systems fail here because the exact answer might not exist in a single sentence in a knowledge base. For example, to answer why a specific organism survives in an ecosystem, a system might need to retrieve several facts about energy transfer and then "reason" through them. The authors aim to bridge this gap by combining the broad search power of IR with the nuanced "understanding" of deep learning models.
Methods
The authors propose a two-stage pipeline:
- Knowledge Retrieval (The "Haystack" Phase): Using a Lucene-based engine, the system searches a massive corpus (Wikipedia, science textbooks, and the ARC Corpus). A key innovation here is term importance weighting: a neural classifier identifies which words in a question are "essential" (e.g., "ocean" in "What is the largest ocean?") and weights them more heavily during the search to ensure the retrieved context is actually relevant.
- Deep Inference (The "Needle" Phase): Once context is found, the system uses an ensemble of "solvers" based on the Bidirectional Attention Flow (BiDAF) architecture. They use two main approaches:
- QA-BiDAF: Directly processes the (Question, Answer, Context) triplet.
- NLI-BiDAF: Converts the question and a candidate answer into an affirmative statement (a "hypothesis") and treats the retrieved context as the "premise." It then predicts if the premise entails, contradicts, or is neutral toward the hypothesis.
Finally, a voting mechanism (a small neural network) combines the scores from the IR engine and the various BiDAF solvers to pick the most likely answer.
Datasets
- Knowledge Bases: English Wikipedia (15GB), ARC Corpus, and CK12 science books.
- Pre-training Sets: To overcome the lack of large-scale science QA data, the authors used Transfer Learning with:
- SQuAD: Modified to turn open-ended questions into multiple-choice.
- SNLI & MultiNLI: Large-scale entailment datasets.
- SciTail: A science-specific entailment dataset.
- Evaluation: The AI2 Reasoning Challenge (ARC) dataset, split into "Easy" and "Challenge" sets.
Key Insights and Ideas
- QA as Entailment: One of the most powerful ideas is reframing multiple-choice QA as a Natural Language Inference (NLI) problem. By turning a question/answer pair into a statement, the model can leverage logic-heavy datasets like MultiNLI to solve science problems.
- The Power of Transfer Learning: Models pre-trained on general-purpose datasets (like image captions in SNLI) surprisingly improved performance on specialized science questions. This suggests that "reasoning" is a transferable skill in NLP.
- IR is for Facts, Inference is for Logic: The results showed that while IR dominates the "Easy" dataset (factoid questions), it struggles on the "Challenge" set. Conversely, the deep inference models provided the most value on the difficult, reasoning-heavy questions.
Key Results
- Accuracy Boost: The two-stage model outperformed the baseline IR solver by 3.08% absolute accuracy on the ARC Challenge test set and 8.47% on the development set.
- Leaderboard Performance: The system ranked 2nd on the ARC Easy leaderboard and 8th on the Challenge leaderboard at the time of writing.
- Synergy: The ensemble model performed better than any single solver, proving that IR and deep inference capture different, complementary aspects of the problem.
Limitations and Future Work
The primary bottleneck identified is the Retrieval Stage. An error analysis revealed that in roughly 50% of the incorrectly answered questions, the IR engine failed to retrieve a context that actually contained the information needed to answer the question. If the "support" text is missing, even the best inference model cannot find the right answer.
Future work suggested by the authors includes:
- Improving the knowledge base with more structured data (like knowledge graphs).
- Developing more sophisticated retrieval methods to find "hidden" context.
- Training on larger, science-specific datasets to reduce the reliance on general-domain pre-training.
Summary updated