Improving Retrieval-Based Question Answering with Deep Inference Models

George-Sebastian Pirtoaca, Traian Rebedea, Stefan Ruseti

Abstract

Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multiple-choice question answering problem. In the first stage, each question-answer pair is fed into an information retrieval engine to find relevant candidate contexts that serve as the underlying knowledge for the inference models. In the second stage, deep learning architectures are used to predict if a candidate answer can be inferred from the context extracted in the first stage. We deploy multiple deep learning architectures pre-trained on different datasets in order to capture semantic features and to enlarge the scope of the questions we can answer correctly. In the end, all these solvers are combined in an ensemble model to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy. Moreover, the model can answer both simple, factoid questions and more complex questions that require reasoning or inference.

Introduction: The Challenge of Non-Factoid QA

While modern AI assistants like Alexa or Google Assistant are excellent at answering "factoid" questions (e.g., "What is the capital of France?"), they struggle with complex, non-factoid questions found in middle-school science exams. These questions often require inference—the ability to connect disparate pieces of information to reach a logical conclusion. Standard retrieval-based systems often fail here because the exact answer might not exist in a single sentence in a knowledge base. This paper aims to bridge this gap by combining the broad search power of Information Retrieval (IR) with the nuanced "understanding" of deep learning inference models.

Methodology: A Two-Stage Pipeline

The authors propose a framework that operates in two distinct stages: Knowledge Retrieval and Deep Inference.

Stage 1: Knowledge Retrieval (The "Haystack" Phase)

Using a Lucene-based engine, the system searches a massive corpus including Wikipedia, science textbooks, and the ARC Corpus. A key innovation is term importance weighting: a neural classifier identifies which words in a question are "essential" (e.g., "ocean" in "What is the largest ocean?") and weights them more heavily during the search to ensure the retrieved context is actually relevant.

Stage 2: Deep Inference (The "Needle" Phase)

Once context is found, the system uses an ensemble of "solvers" based on the Bidirectional Attention Flow (BiDAF) architecture. These solvers either process the (Question, Answer, Context) triplet directly or reframe the task as an entailment problem. In the latter approach, the question and candidate answer are converted into a hypothesis statement, and the model predicts if the retrieved context (the premise) entails, contradicts, or is neutral toward it.

The deep learning architecture used for scoring question-answer pairs, integrating BiDAF and biLSTM layers.

Datasets and Transfer Learning

To overcome the lack of large-scale science QA data, the authors leverage Transfer Learning. Models are pre-trained on diverse datasets including SQuAD (modified for multiple-choice), SNLI/MultiNLI (general entailment), and SciTail (science-specific entailment). Evaluation is performed on the AI2 Reasoning Challenge (ARC) dataset, split into "Easy" and "Challenge" sets.

Key Results and Insights

The two-stage model demonstrates significant improvements over traditional retrieval-based methods, particularly on reasoning-heavy questions. Reframing QA as entailment allows the model to leverage logic-heavy datasets like MultiNLI to solve science problems.

Performance comparison on the ARC Challenge dataset, showing the improvement of the two-stage model over various baselines.

As shown in the results, the two-stage model outperformed the baseline IR solver by 3.08% absolute accuracy on the ARC Challenge test set and 8.47% on the development set. The ensemble model performed better than any single solver, proving that IR and deep inference capture different, complementary aspects of the problem: IR is effective for facts, while inference is essential for logic.

Limitations and Future Work

The primary bottleneck identified is the Retrieval Stage. Error analysis revealed that in roughly 50% of incorrect answers, the IR engine failed to retrieve a context containing the necessary information. Future work includes improving the knowledge base with structured data (like knowledge graphs) and developing more sophisticated retrieval methods to find "hidden" context.

Summary updated

Abstract

Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex science questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multi-choice question answering in the science domain. For each question-answer pair, we use standard retrieval-based models to find relevant candidate contexts and decompose the main problem into two different sub-problems. First, assign correctness scores for each candidate answer based on the context using retrieval models from Lucene. Second, we use deep learning architectures to compute if a candidate answer can be inferred from some well-chosen context consisting of sentences retrieved from the knowledge base. In the end, all these solvers are combined using a simple neural network to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy.
BibTeX
@inproceedings{pirtoaca2019improving,
  author = {George-Sebastian Pirtoaca and Traian Rebedea and Stefan Ruseti},
  title = {{Improving Retrieval-Based Question Answering with Deep Inference Models}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=eXmh0tR65b7},
}
Figure