Abstract
Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multiple-choice question answering problem. In the first stage, each question-answer pair is fed into an information retrieval engine to find relevant candidate contexts that serve as the underlying knowledge for the inference models. In the second stage, deep learning architectures are used to predict if a candidate answer can be inferred from the context extracted in the first stage. We deploy multiple deep learning architectures pre-trained on different datasets in order to capture semantic features and to enlarge the scope of the questions we can answer correctly. In the end, all these solvers are combined in an ensemble model to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy. Moreover, the model can answer both simple, factoid questions and more complex questions that require reasoning or inference.
Introduction: The Challenge of Non-Factoid QA
While modern AI assistants like Alexa or Google Assistant are excellent at answering "factoid" questions (e.g., "What is the capital of France?"), they struggle with complex, non-factoid questions found in middle-school science exams. These questions often require inference—the ability to connect disparate pieces of information to reach a logical conclusion. Standard retrieval-based systems often fail here because the exact answer might not exist in a single sentence in a knowledge base. This paper aims to bridge this gap by combining the broad search power of Information Retrieval (IR) with the nuanced "understanding" of deep learning inference models.
Methodology: A Two-Stage Pipeline
The authors propose a framework that operates in two distinct stages: Knowledge Retrieval and Deep Inference.
Stage 1: Knowledge Retrieval (The "Haystack" Phase)
Using a Lucene-based engine, the system searches a massive corpus including Wikipedia, science textbooks, and the ARC Corpus. A key innovation is term importance weighting: a neural classifier identifies which words in a question are "essential" (e.g., "ocean" in "What is the largest ocean?") and weights them more heavily during the search to ensure the retrieved context is actually relevant.
Stage 2: Deep Inference (The "Needle" Phase)
Once context is found, the system uses an ensemble of "solvers" based on the Bidirectional Attention Flow (BiDAF) architecture. These solvers either process the (Question, Answer, Context) triplet directly or reframe the task as an entailment problem. In the latter approach, the question and candidate answer are converted into a hypothesis statement, and the model predicts if the retrieved context (the premise) entails, contradicts, or is neutral toward it.
Datasets and Transfer Learning
To overcome the lack of large-scale science QA data, the authors leverage Transfer Learning. Models are pre-trained on diverse datasets including SQuAD (modified for multiple-choice), SNLI/MultiNLI (general entailment), and SciTail (science-specific entailment). Evaluation is performed on the AI2 Reasoning Challenge (ARC) dataset, split into "Easy" and "Challenge" sets.
Key Results and Insights
The two-stage model demonstrates significant improvements over traditional retrieval-based methods, particularly on reasoning-heavy questions. Reframing QA as entailment allows the model to leverage logic-heavy datasets like MultiNLI to solve science problems.
As shown in the results, the two-stage model outperformed the baseline IR solver by 3.08% absolute accuracy on the ARC Challenge test set and 8.47% on the development set. The ensemble model performed better than any single solver, proving that IR and deep inference capture different, complementary aspects of the problem: IR is effective for facts, while inference is essential for logic.
Limitations and Future Work
The primary bottleneck identified is the Retrieval Stage. Error analysis revealed that in roughly 50% of incorrect answers, the IR engine failed to retrieve a context containing the necessary information. Future work includes improving the knowledge base with structured data (like knowledge graphs) and developing more sophisticated retrieval methods to find "hidden" context.
Summary updated