Parameter-Efficient Neural Question Answering Models via Graph-Enriched Document Representations

Louis Castricato, Stephen Fitz, Won Young ShinarXiv 2021 · OpenReview

Abstract

As the computational footprint of modern NLP systems grows, it becomes increasingly important to arrive at more efficient models. We show that by employing graph convolutional document representation, we can arrive at a question answering system that performs comparably to, and in some cases exceeds the SOTA solutions, while using less than 5% of their resources in terms of trainable parameters. As it currently stands, a major issue in applying GCNs to NLP is document representation. In this paper, we show that a GCN enriched document representation greatly improves the results seen in HotPotQA, even when using a trivial topology. Our model (gQA), performs admirably when compared to the current SOTA, and requires little to no preprocessing. In Shao et al. 2020, the authors suggest that graph networks are not necessary for good performance in multi-hop QA. In this paper, we suggest that large language models are not necessary for good performance by showing a naïve implementation of a GCN performs comparably to SoTA models based on pretrained language models.

Introduction

The current trajectory of Natural Language Processing (NLP) is defined by a "bigger is better" philosophy. To tackle complex tasks like multi-hop question answering—where a system must synthesize information across multiple documents—researchers typically deploy massive pre-trained models like RoBERTa or BERT. These models often contain hundreds of millions or even billions of parameters, demanding significant computational resources for both training and inference.

This paper introduces gQA, a graph-based question-answering model that challenges the necessity of these "Goliaths." By leveraging Gated Graph Neural Networks (GGNNs) to represent document structure, gQA achieves performance comparable to State-of-the-Art (SOTA) models while utilizing less than 5% of the trainable parameters.

The gQA Architecture

The core innovation of gQA lies in its parameter-efficient document representation. Instead of treating text as a simple linear sequence, the model represents documents as a graph of sentences.

The gQA architecture consists of an encoder that builds graph-based representations and a decoder that predicts answer spans and types.

The architecture is divided into two main components:

  1. Encoder: Sentences are first encoded individually using a BiLSTM and CharCNN with GloVe embeddings. The document is then structured as a graph where each sentence is a node. An adjacency matrix $M$ defines connections based on sequential order, content similarity (cosine similarity), and structural links (connecting the end of one document to the start of others). A GGNN then performs message passing across this topology to capture inter-sentence relationships.
  2. Decoder: The decoder uses a series of BiLSTMs and linear classifiers to predict supporting facts, the start and end indices of the answer string, and the answer type (e.g., yes/no or entity).

Experimental Results

The model was evaluated on the HotPotQA dataset, a benchmark for multi-hop reasoning. The authors compared the gQA variants against several high-performing models that rely on large-scale pre-training.

Comparison of gQA variants against SOTA models. gQA-S achieves competitive results with a fraction of the parameters.

As shown in the table above, the gQA-S model, with only 9.04 million parameters, achieved an Exact Match (EM) score of 68.47. This outperformed the much larger HGN (125M+ parameters) and DFGN (110M+ parameters) models in the EM metric. While its F1 score (76.85) was competitive with HGN's (79.36), it achieved this with less than 1/10th of the parameters and without any large-scale language model pre-training.

Key Insights

The success of gQA suggests that structural representation can be more impactful than raw scale. By explicitly defining how sentences relate to one another through a graph topology, the model does not need to learn these relationships from scratch using billions of parameters. Even a "trivial" topology—combining sequence and similarity—provided a significant performance boost over the baseline.

Furthermore, this approach promotes sustainability in AI. Training the gQA-S model took only 12 hours on a single mid-range GPU (GTX 1080), representing a massive reduction in the carbon footprint and financial cost associated with training SOTA QA systems.

Conclusion and Future Work

The gQA model demonstrates that graph-enriched representations are a powerful alternative to massive Transformers for complex reasoning tasks. While the current implementation avoids pre-trained language models to highlight the strength of the graph approach, future work could explore combining these two paradigms. Additionally, investigating more sophisticated pooling mechanisms and refining the graph construction process could further close the gap with the largest SOTA models.

Summary updated

Abstract

As the computational footprint of modern NLP systems grows, it becomes increasingly important to arrive at more efficient models. We show that by employing graph convolutional document representation, we can arrive at a question answering system that performs comparably to, and in some cases exceeds the SOTA solutions, while using less than 5\% of their resources in terms of trainable parameters. As it currently stands, a major issue in applying GCNs to NLP is document representation. In this paper, we show that a GCN enriched document representation greatly improves the results seen in HotPotQA, even when using a trivial topology. Our model (gQA), performs admirably when compared to the current SOTA, and requires little to no preprocessing. In Shao et al. 2020, the authors suggest that graph networks are not necessary for good performance in multi-hop QA. In this paper, we suggest that large language models are not necessary for good performance by showing a na\"{i}ve implementation of a GCN performs comparably to SoTA models based on pretrained language models.
BibTeX
@misc{castricato2021parameter,
  author = {Louis Castricato and Stephen Fitz and Won Young Shin},
  title = {{Parameter-Efficient Neural Question Answering Models via Graph-Enriched Document Representations}},
  year = {2021},
  eprint = {2106.00851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.00851},
}
Figure