Skoltech at TextGraphs-17 Shared Task: Finding GPT-4 Prompting Strategies for Multiple Choice Questions

Maria Lysyuk, Pavel BraslavskiACL 2024

TL;DR

This paper tackles the challenge of using large language models (LLMs) like GPT-4 for multiple-choice question answering on knowledge graphs. The authors developed a system that leverages GPT-4 with chain-of-thought prompting, enhanced by graph subgraph features and Wikidata descriptions, achieving a strong second-place finish in the TextGraphs-17 shared task. Their work demonstrates how to effectively combine LLMs with structured knowledge for improved accuracy, especially in cases with ambiguous or identical answer choices.

What Problem Does This Paper Address?

The paper addresses the problem of accurately answering multiple-choice questions where the answers are linked to knowledge graphs. Specifically, the TextGraphs-17 shared task involves selecting the correct entity from a list of candidates, each associated with a Wikidata subgraph representing its connection to the question's entities. Two key challenges arise: 1) questions can have multiple correct answers, and 2) a significant portion of candidate answers share identical textual labels, making it difficult for LLMs to distinguish the correct one based on text alone. This research aims to develop a robust system that can overcome these ambiguities by integrating LLM reasoning with graph structural information and external knowledge.

Methods

The core of the proposed method is GPT-4 with chain-of-thought (CoT) prompting. The baseline system uses GPT-4 to generate an answer through step-by-step reasoning and then verifies if this answer is present in the provided candidate list. To improve upon this baseline, several enhancements were introduced:

  • Subgraph Size as a Feature: The authors observed that correct answers often correspond to smaller subgraphs connecting the question entities to the answer entity. This structural information is used to help disambiguate answers.
  • Wikidata Answer Descriptions: For candidate answers with identical textual labels, their Wikidata descriptions are used as additional context. This allows the model to differentiate between entities that might appear the same on the surface but have distinct meanings or roles.
  • Question Rephrasing: To address questions that the baseline fails to answer, a technique of rephrasing the question is employed to provide more context and clarity, making it easier for the LLM to process.
  • Fuzzy Answer Matching: A fuzzy string matching approach is used to handle minor variations in the surface form of answers between the LLM's output and the candidate list.
  • Handling Identical Labels: When multiple candidate answers have identical labels, the system prioritizes entities with non-empty Wikidata descriptions. If ambiguity persists, it uses fastText embeddings to find the most semantically similar candidate to the question.

Datasets

The primary dataset used for training and evaluation is the TextGraphs-17 Shared Task dataset. This dataset is based on the Mintaka dataset (Sen et al., 2022), which provides questions, candidate answer entities, and associated Wikidata subgraphs. The paper notes that the Mintaka dataset includes various question types (superlative, difference, multi-hop, etc.), although the focus of their solution was on handling identical answer labels and unanswered questions.

Key Insights and Ideas

  • Subgraph Structure is Informative: The size and structure of the knowledge graph subgraph connecting question entities to candidate answers provide valuable signals for disambiguation, especially when textual clues are insufficient. Smaller subgraphs are more likely to lead to correct answers.
  • External Knowledge (Wikidata Descriptions) is Crucial: For cases with identical textual labels, leveraging Wikidata descriptions is essential for distinguishing between semantically similar but factually different entities.
  • LLMs Benefit from Structured Context: Simply relying on LLM reasoning alone is not enough for complex tasks like KGQA. Augmenting LLMs with structured graph data and external knowledge significantly boosts performance.
  • Prompt Engineering is Key for Robustness: Techniques like question rephrasing and providing candidate descriptions within the prompt are effective strategies for improving the LLM's ability to handle difficult or ambiguous questions.

Key Results

The proposed method achieved an F1 score of 0.83, securing 2nd place in the TextGraphs-17 shared task. This represents a 0.05 improvement over the GPT-4 baseline (F1 score of 0.78). The paper highlights that the most significant improvements came from addressing the challenge of differentiating identical answer labels. An ablation study showed that the developed tricks also improved the performance of an open-source LLM (Llama-3-8B-Instruct), though the baseline performance of Llama was lower.

Limitations and Future Work

  • Proprietary LLM: The primary model used is GPT-4, which is a proprietary LLM. The authors conducted an ablation study with an open-source LLM (Llama-3-8B-Instruct) to assess transferability, but further research with various open-source models could be beneficial.
  • Unanswered Questions: Despite improvements, a small percentage of questions (24 out of 1,000) remained unanswered by the baseline and post-processing steps, requiring additional tricks.
  • Future Work: The paper suggests that further exploration of question types from the Mintaka dataset could lead to more specialized prompting strategies. Additionally, investigating how to better utilize the graph structure and potentially explore more complex graph reasoning techniques could be fruitful avenues for future research. The authors also mention that some ideas tried, like using question types for few-shot prompting or modifying questions to check answers individually, did not work and could be revisited with different approaches.

Summary updated

BibTeX
@inproceedings{lysyuk2024skoltech,
  author = {Maria Lysyuk and Pavel Braslavski},
  title = {{Skoltech at TextGraphs-17 Shared Task: Finding GPT-4 Prompting Strategies for Multiple Choice Questions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure