Abstract
As the computational footprint of modern NLP systems grows, it becomes increasingly important to arrive at more efficient models. We show that by employing graph convolutional document representation, we can arrive at a question answering system that performs comparably to, and in some cases exceeds the SOTA solutions, while using less than 5% of their resources in terms of trainable parameters. As it currently stands, a major issue in applying GCNs to NLP is document representation. In this paper, we show that a GCN enriched document representation greatly improves the results seen in HotPotQA, even when using a trivial topology. Our model (gQA), performs admirably when compared to the current SOTA, and requires little to no preprocessing. In Shao et al. 2020, the authors suggest that graph networks are not necessary for good performance in multi-hop QA. In this paper, we suggest that large language models are not necessary for good performance by showing a naïve implementation of a GCN performs comparably to SoTA models based on pretrained language models.
Introduction
The current trajectory of Natural Language Processing (NLP) is defined by a "bigger is better" philosophy. To tackle complex tasks like multi-hop question answering—where a system must synthesize information across multiple documents—researchers typically deploy massive pre-trained models like RoBERTa or BERT. These models often contain hundreds of millions or even billions of parameters, demanding significant computational resources for both training and inference.
This paper introduces gQA, a graph-based question-answering model that challenges the necessity of these "Goliaths." By leveraging Gated Graph Neural Networks (GGNNs) to represent document structure, gQA achieves performance comparable to State-of-the-Art (SOTA) models while utilizing less than 5% of the trainable parameters.
The gQA Architecture
The core innovation of gQA lies in its parameter-efficient document representation. Instead of treating text as a simple linear sequence, the model represents documents as a graph of sentences.
The architecture is divided into two main components:
- Encoder: Sentences are first encoded individually using a BiLSTM and CharCNN with GloVe embeddings. The document is then structured as a graph where each sentence is a node. An adjacency matrix $M$ defines connections based on sequential order, content similarity (cosine similarity), and structural links (connecting the end of one document to the start of others). A GGNN then performs message passing across this topology to capture inter-sentence relationships.
- Decoder: The decoder uses a series of BiLSTMs and linear classifiers to predict supporting facts, the start and end indices of the answer string, and the answer type (e.g., yes/no or entity).
Experimental Results
The model was evaluated on the HotPotQA dataset, a benchmark for multi-hop reasoning. The authors compared the gQA variants against several high-performing models that rely on large-scale pre-training.
As shown in the table above, the gQA-S model, with only 9.04 million parameters, achieved an Exact Match (EM) score of 68.47. This outperformed the much larger HGN (125M+ parameters) and DFGN (110M+ parameters) models in the EM metric. While its F1 score (76.85) was competitive with HGN's (79.36), it achieved this with less than 1/10th of the parameters and without any large-scale language model pre-training.
Key Insights
The success of gQA suggests that structural representation can be more impactful than raw scale. By explicitly defining how sentences relate to one another through a graph topology, the model does not need to learn these relationships from scratch using billions of parameters. Even a "trivial" topology—combining sequence and similarity—provided a significant performance boost over the baseline.
Furthermore, this approach promotes sustainability in AI. Training the gQA-S model took only 12 hours on a single mid-range GPU (GTX 1080), representing a massive reduction in the carbon footprint and financial cost associated with training SOTA QA systems.
Conclusion and Future Work
The gQA model demonstrates that graph-enriched representations are a powerful alternative to massive Transformers for complex reasoning tasks. While the current implementation avoids pre-trained language models to highlight the strength of the graph approach, future work could explore combining these two paradigms. Additionally, investigating more sophisticated pooling mechanisms and refining the graph construction process could further close the gap with the largest SOTA models.
Summary updated