nlp_enjoyers at TextGraphs-17 Shared Task: Text-Graph Representations for Knowledge Graph Question Answering using all-MPNet

Nikita Kurdiukov, Viktoriia Zinkovich, Sergey Karpukhin, Pavel TikhomirovACL 2024

TL;DR

This paper tackles the challenge of Multiple Choice Question Answering (MCQA) by integrating knowledge graph (KG) information with powerful language models. They propose a method to linearize KG subgraphs and feed them directly into the prompt, significantly improving performance, especially with the MPNet architecture.

What Problem Does This Paper Address?

The paper addresses the growing need for efficient information retrieval and question answering systems, particularly focusing on Multiple Choice Question Answering (MCQA). While standard NLP models like BERT and its variants excel at many tasks, MCQA presents a unique challenge. It requires not just understanding the text but also making logical inferences and leveraging external knowledge. Existing approaches often struggle because they might prioritize grammatical correctness over logical coherence, leading to incorrect answers. This work aims to bridge this gap by explicitly incorporating structured knowledge from knowledge graphs into the answering process.

Methods

The core of the proposed method involves combining textual information with structured data from knowledge graphs. Specifically, they extract relevant subgraphs from a knowledge graph (Wikidata) that connect entities mentioned in the question to candidate answers. These subgraphs are then "linearized" into a textual format, essentially converting the graph structure into a sequence of tokens. This linearized graph representation is then appended to the original question prompt, separated by special tokens, and fed into a language model.

They experimented with several language model backbones, including MiniLM and T5, but found the MPNet architecture to be particularly effective. To optimize training and handle the large scale of the data, they employed LoRA (Low-Rank Adaptation) for efficient fine-tuning and an oversampling technique to address class imbalance in the dataset. The model is trained as a binary classifier, predicting the likelihood of each candidate answer being correct, and the highest-scoring answer is selected.

Datasets

The primary dataset used in this research is the TextGraphs-17 shared-task dataset. This dataset comprises 37,672 question-answer pairs, annotated with Wikidata entities. It includes information about entities present in both questions and answers, as well as pre-computed shortest-path subgraphs connecting question entities to candidate answer entities within the Wikidata knowledge graph.

Key Insights and Ideas

  • Linearized Subgraphs are Powerful: Directly incorporating linearized knowledge graph subgraphs into the prompt significantly boosts the performance of language models on MCQA tasks. This suggests that models can effectively leverage structured relational information when presented in a textual format.
  • Architecture Matters: The choice of language model architecture plays a crucial role. MPNet, an encoder-based model, demonstrated superior performance compared to other tested architectures like MiniLM and T5.
  • LoRA for Efficiency: Fine-tuning large language models with LoRA proved to be an efficient method for adapting them to the MCQA task, allowing for effective training without the prohibitive computational cost of full fine-tuning.
  • Graph Context is Key: The study confirms that knowledge graphs provide valuable context that improves a model's ability to answer questions, moving beyond purely text-based reasoning.

Key Results

The paper reports the F1 scores achieved on the public test set. The best-performing model, all-MPNet, utilizing linearized subgraphs, achieved an F1 score of 0.3887. This significantly outperformed models without graph information or those using different architectures. For instance, the all-MiniLM model achieved an F1 score of 0.3463 with linearized graphs, and a T5-Small model achieved 0.3180. An ablation study also showed that increasing the sequence length (from 256 to 512) further improved the F1 score for the all-MiniLM model when using linearized graphs, reaching 0.3463.

Limitations and Future Work

The authors acknowledge a significant limitation: their reliance on pre-existing subgraphs or triplets derived from the graph. This means they did not explore more advanced subgraph sampling or generation strategies, which could potentially yield even better results.

Future work could involve investigating these more dynamic subgraph extraction methods. Additionally, the paper highlights the untapped potential of in-context learning and prompting strategies with decoder-based architectures. Exploring these techniques, given their current popularity and effectiveness, is identified as a promising direction for future research.

Summary updated

BibTeX
@inproceedings{kurdiukov2024nlp,
  author = {Nikita Kurdiukov and Viktoriia Zinkovich and Sergey Karpukhin and Pavel Tikhomirov},
  title = {{nlp\_enjoyers at TextGraphs-17 Shared Task: Text-Graph Representations for Knowledge Graph Question Answering using all-MPNet}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure