GNN is a Counter? Revisiting GNN for Question Answering

Kuan Wang, Yuyu Zhang, Diyi Yang, Le Song, Tao Qin

Abstract

Question Answering (QA) has been a long-standing research topic in AI and NLP fields, and a wealth of studies have been conducted to attempt to equip QA systems with human-level reasoning capability. To approximate the complicated human reasoning process, state-of-the-art QA systems commonly use pre-trained language models (LMs) to access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs). However, many problems remain open regarding the reasoning functionality of these GNN-based modules. Can these GNN-based modules really perform a complex reasoning process? Are they under- or over-complicated for QA? To open the black box of GNN and investigate these problems, we dissect state-of-the-art GNN modules for QA and analyze their reasoning capability. We discover that even a very simple graph neural counter can outperform all the existing GNN modules on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. Our work reveals that existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting. It remains a challenging open problem to build comprehensive reasoning modules for knowledge-powered QA.

Introduction

Modern Question Answering (QA) systems aim to replicate human reasoning by combining the implicit knowledge of large language models (LMs) with structured knowledge from Knowledge Graphs (KGs). Typically, Graph Neural Networks (GNNs) are employed to perform multi-hop reasoning over these graphs. However, the internal mechanics of these GNN modules remain a "black box." This research investigates whether these complex architectures are truly necessary or if they are over-engineered for the tasks at hand.

Methodology: Dissection and Redesign

The researchers employed a two-stage approach to analyze and simplify GNN-based QA. First, they used Sparse Variational Dropout (SparseVD) as a diagnostic tool to dissect state-of-the-art models like QA-GNN and MHGRN. By observing which parameters could be pruned to zero without impacting performance, they identified significant redundancies in node embeddings and message-passing layers.

The SparseVD dissection process identifies redundant GNN components by analyzing node embeddings, adjacency matrices, and edge embeddings to determine which ingredients are crucial for reasoning.

Second, based on these findings, they developed the Graph Soft Counter (GSC). The GSC is a minimal module with a hidden dimension of exactly one. It replaces high-dimensional message passing with the propagation of single scalar values, effectively performing "soft counts" of nodes and edges. The model was evaluated on CommonsenseQA and OpenBookQA using ConceptNet as the external knowledge source.

Key Insights and Results

The dissection revealed that GNNs in QA are massively over-parameterized. Surprisingly, initial node embeddings—often considered crucial—were found to be dispensable. Instead, the type of relationship (edge type) between concepts proved to be the most vital information for the graph module.

The GSC achieved state-of-the-art results while being significantly more efficient:

  • Extreme Parameter Reduction: GSC uses only 3,000 learnable parameters, a 99.9% reduction compared to the 2.8 million in QA-GNN.
  • Performance Benchmarks: It outperformed existing GNN-based systems on CommonsenseQA (76.2% test accuracy) and reached 87.4% on OpenBookQA, beating massive models like UnifiedQA (11B parameters) and T5 (3B parameters), which are 30x and 8x larger than the GSC system, respectively.
  • High Interpretability: Since the model operates on scalars, the reasoning process is transparent; researchers can directly inspect which edges contributed most to an answer prediction.
The proposed pipeline simplifies reasoning to a counting process, demonstrating that simple edge counting can be more effective, efficient, and interpretable than complex GNN architectures.

Conclusion

The success of the Graph Soft Counter suggests that current "knowledge-aware reasoning" in QA benchmarks is largely a matter of sophisticated counting rather than complex logical inference. While GSC provides a highly efficient and interpretable baseline, the findings highlight a significant gap in the field: existing benchmarks may not yet require the deep, multi-hop reasoning that complex GNNs were designed to provide. Developing modules capable of true human-level logical inference remains a critical open challenge.

Summary updated

Abstract

Question Answering (QA) has been a long-standing research topic in AI and NLP fields, and a wealth of studies have been conducted to attempt to equip QA systems with human-level reasoning capability. To approximate the complicated human reasoning process, state-of-the-art QA systems commonly use pre-trained language models (LMs) to access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs). However, many problems remain open regarding the reasoning functionality of these GNN-based modules. Can these GNN-based modules really perform a complex reasoning process? Are they under- or over-complicated for QA? To open the black box of GNN and investigate these problems, we dissect state-of-the-art GNN modules for QA and analyze their reasoning capability. We discover that even a very simple graph neural counter can outperform all the existing GNN modules on CommonsenseQA and OpenBookQA, two popular QA benchmark datasets which heavily rely on knowledge-aware reasoning. Our work reveals that existing knowledge-aware GNN modules may only carry out some simple reasoning such as counting. It remains a challenging open problem to build comprehensive reasoning modules for knowledge-powered QA.
BibTeX
@inproceedings{wang2022gnn,
  author = {Kuan Wang and Yuyu Zhang and Diyi Yang and Le Song and Tao Qin},
  title = {{GNN is a Counter? Revisiting GNN for Question Answering}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=hzmQ4wOnSb},
}
Figure