BigText-QA: Question Answering over a Large-Scale Hybrid Knowledge Graph

Jingjing Xu, Maria Biryukov, Martin Theobald, Vinu Ellampallil Venugopal

TL;DR

This paper introduces BigText-QA, a novel question-answering system that leverages a "hybrid knowledge graph" (KG) combining structured knowledge from traditional KBs with rich contextual information from unstructured text. BigText-QA demonstrates competitive performance against state-of-the-art systems, particularly excelling in handling complex questions by integrating diverse information sources.

What Problem Does This Paper Address?

Answering complex questions that require understanding nuanced relationships between multiple entities is a significant challenge in natural language processing. While existing knowledge bases (KBs) like DBpedia and Wikidata provide structured facts, they often lack the contextual depth found in natural language. Conversely, information retrieval (IR) systems excel at handling large text collections but struggle with structured reasoning. This paper addresses the gap by proposing a system that bridges these two worlds, creating a unified representation that captures both structured facts and their rich textual context.

Methods

The core of BigText-QA is its BigText Knowledge Graph (BigText-KG). This KG is a "hybrid" representation that goes beyond traditional triple-based KBs by:

  • Integrating Structured and Unstructured Data: It combines a canonical set of named entities mapped to a background KB (like YAGO or Wikidata) with an open set of textual clauses.
  • Preserving Document Structure: The KG retains original document structure, domains, hyperlinks, and metadata, decomposing them into sentences, clauses, and mentions.
  • Capturing Relational Diversity: Textual clauses provide "highly diversified relational paraphrases with rich context," offering a more nuanced understanding of relationships than simple triples.
  • Leveraging Syntactic and Semantic Dependencies: The system uses syntactic dependencies and co-reference resolution to establish links between entities and their contexts, even across document boundaries.
  • Property Graph Model: BigText-KG is implemented as a property graph, distinguishing five vertex types (documents, sentences, clauses, mentions, entities) and associating extensible properties with each.
  • Question Answering Pipeline: BigText-QA translates natural language questions into a "quasi-graph" representation, similar to the QUEST system, and then uses a Group Steiner Tree (GST) algorithm to find answers. Vertex and edge weights are crucial for this process, derived from similarities between question terms and graph components.

Datasets

The paper evaluates BigText-QA on two benchmark datasets:

  • CQ-W: A dataset of complex questions derived from Wiki-Answers. The authors filtered out questions whose answers were not present in their Wikipedia-based BigText subgraph (approximately 25% of questions).
  • TriviaQA: A large-scale dataset of complex and compositional questions. For evaluation, the authors randomly selected 79 questions from the development set.

The BigText-KG itself is built using a large Wikipedia dump from 2019.

Key Insights and Ideas

  • Hybrid KG for Contextual Depth: The fundamental insight is that combining structured KBs with rich textual context in a unified graph representation is crucial for answering complex questions that require nuanced understanding.
  • Redundancy as a Feature: The BigText-KG is intentionally "redundant," incorporating multiple representations of knowledge (structured facts and textual paraphrases) to enhance its coverage and accuracy.
  • Syntactic and Semantic Links are Key: The paper emphasizes the importance of syntactic dependencies and co-reference resolution in disambiguating entities and establishing meaningful connections within the KG.
  • Compactness and Density Matter: The results suggest that generating more compact yet denser quasi-graphs (through careful thresholding of alignment edges) can lead to better QA performance.

Key Results

  • BigText-QA achieves competitive results with QUEST, a graph-based unsupervised QA system, and outperforms DrQA, a state-of-the-art neural-network-based QA system, on complex questions.
  • The system demonstrates strong performance on both the CQ-W and TriviaQA datasets, with Mean Reciprocal Rank (MRR) scores often comparable to or exceeding QUEST, especially when using optimized edge thresholds.
  • BigText-QA generally generates more compact yet denser graphs compared to QUEST, which positively impacts the answer candidate selection.
  • The paper highlights that different NLP preprocessing choices (e.g., sentence decomposition) can lead to varying quasi-graph configurations, impacting performance.

Limitations and Future Work

  • Movie-Related Questions: The paper notes that BigText-QA does not perform as favorably on "Movie"-related questions within the CQ-W dataset, suggesting this as an area for future investigation.
  • Threshold Sensitivity: The performance is sensitive to the chosen thresholds for alignment edges. The paper observes that thresholds below 0.5 have a negligible effect, while increasing it from 0.5 to 0.6 significantly reduces the number of edges, potentially due to the word2vec model's similarity estimations.
  • Future Work: The authors suggest further investigation into the performance on specific question categories and exploring more advanced NLP techniques to further refine the KG construction and QA process.

Summary updated

BibTeX
@inproceedings{xu2023bigtext,
  author = {Jingjing Xu and Maria Biryukov and Martin Theobald and Vinu Ellampallil Venugopal},
  title = {{BigText-QA: Question Answering over a Large-Scale Hybrid Knowledge Graph}},
  booktitle = {BDTA},
  year = {2023},
  doi = {10.1007/978-3-031-52265-9\_3},
}
Figure