Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering

Sewon Min, Danqi Chen, Luke Zettlemoyer, Hannaneh HajishirziarXiv 2019 · arXiv

Introduction

Open-domain question answering (QA) systems typically rely on two distinct types of knowledge sources: structured Knowledge Bases (KBs) like Wikidata and unstructured text corpora like Wikipedia. While KBs provide precise, structured facts, they often suffer from incompleteness. Conversely, text corpora offer massive coverage but are difficult to reason over algorithmically. Standard "retrieve-and-read" systems often treat retrieved passages in isolation, which leads to failure when an answer requires synthesizing information across multiple documents or disambiguating entities based on context.

This research introduces a novel framework that bridges these two worlds. By representing retrieved text passages as a graph connected by KB relations, the system can identify relevant evidence that keyword-based methods miss and "read" across multiple passages to solve complex queries. This approach demonstrates significant performance gains, outperforming standard text-only models by up to 11%.

The Challenge of Keyword Matching and Isolated Reading

Traditional QA systems face a "keyword matching" bottleneck. If a question does not share specific words with the ideal evidence passage, standard retrieval methods like TF-IDF often fail to find it. Furthermore, standard readers process passages independently, which can lead to errors in disambiguation. For example, a model might identify a person mentioned in a song's credits as the answer to a question about a singer, without realizing that a related passage about the album provides the necessary context to disqualify that candidate.

An example of cross-passage reasoning: distinguishing between a song and an album to identify the correct vocalist.

Methodology

The proposed framework consists of two primary components: GraphRetriever and GraphReader.

GraphRetriever

Instead of returning a flat list of passages, GraphRetriever constructs a structured graph. It begins with "seed" passages identified through traditional keyword search and entity linking. It then expands this set by traversing edges in a Knowledge Base (Wikidata) and following "inner-document" links (e.g., other passages within the same Wikipedia article). This expansion allows the system to discover highly relevant text that may not contain any direct keywords from the original query.

GraphReader

While most readers process each passage in a vacuum, GraphReader utilizes a BERT-based architecture enhanced with graph-aware fusion layers. These layers employ a Graph Convolutional Network (GCN) approach to propagate information between passages along the edges of the constructed graph. This ensures that the representation of any single passage is informed by its neighbors before a final answer is predicted.

The overall architecture: GraphRetriever constructs a passage graph using KB relations, and GraphReader uses fusion layers to update representations across the graph.

Experimental Results

The model was evaluated on three major open-domain QA benchmarks: WebQuestions, Natural Questions, and TriviaQA. Using Wikipedia as the text source and Wikidata as the KB, the graph-based approach consistently outperformed non-graph baselines by 2-11% in Exact Match (EM) scores.

Key findings from the evaluation include:

  • Early Fusion Benefits: Using the KB to guide retrieval allows the system to find evidence that keyword-based methods miss entirely.
  • Structural Shortcuts: KB relations act as shortcuts between distant pieces of text. For instance, a "parent organization" link can connect a subsidiary's history to its parent company's founding date, even if the information resides in different Wikipedia articles.
  • Relational Awareness: While even a simple binary indicator of a relationship between passages improves performance, explicitly modeling the specific type of relation (e.g., "performer" or "is located in") yields the best results.

Conclusion and Future Work

This work demonstrates that structured knowledge can significantly enhance the retrieval and reading of unstructured text for open-domain QA. By leveraging the connectivity of Knowledge Bases, the system achieves state-of-the-art results without the need for massive, end-to-end pre-training.

Future research may address current limitations, such as the memory intensity of training on full passage graphs and the dependency on the coverage of the underlying KB. Additionally, exploring joint end-to-end training of the retriever and reader could provide further performance improvements, albeit at a higher computational cost.

Summary updated

Abstract

We introduce an approach for open-domain question answering (QA) that retrieves and reads a passage graph, where vertices are passages of text and edges represent relationships that are derived from an external knowledge base or co-occurrence in the same article. Our goals are to boost coverage by using knowledge-guided retrieval to find more relevant passages than text-matching methods, and to improve accuracy by allowing for better knowledge-guided fusion of information across related passages. Our graph retrieval method expands a set of seed keyword-retrieved passages by traversing the graph structure of the knowledge base. Our reader extends a BERT-based architecture and updates passage representations by propagating information from related passages and their relations, instead of reading each passage in isolation. Experiments on three open-domain QA datasets, WebQuestions, Natural Questions and TriviaQA, show improved performance over non-graph baselines by 2-11% absolute. Our approach also matches or exceeds the state-of-the-art in every case, without using an expensive end-to-end training regime.
BibTeX
@misc{min2019knowledge,
  author = {Sewon Min and Danqi Chen and Luke Zettlemoyer and Hannaneh Hajishirzi},
  title = {{Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering}},
  year = {2019},
  eprint = {1911.03868},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.03868},
}
Figure