BITS Pilani at SemEval-2024 Task 9: Prompt Engineering with GPT-4 for Solving Brainteasers

Dilip Venkatesh, Yashvardhan SharmaNAACL 2024

TL;DR

This paper explores using OpenAI's GPT-4 model with prompt engineering to solve brainteasers, a task requiring complex, often "out-of-the-box" reasoning. The authors demonstrate that a well-crafted prompt can effectively guide GPT-4 to achieve competitive results on the SemEval-2024 BRAINTEASER task, highlighting the power of prompt engineering for complex reasoning challenges.

What Problem Does This Paper Address?

The paper tackles the challenge of solving "brainteasers," which are puzzles that often defy common sense and require lateral thinking rather than purely linear, logical deduction. While large language models (LLMs) have shown impressive capabilities in traditional reasoning tasks, their ability to handle these unconventional, creative problem-solving scenarios is less explored. The SemEval-2024 BRAINTEASER task specifically aims to test this by presenting sentence and word puzzles that require understanding beyond literal meaning, often involving wordplay or unexpected interpretations. The task also includes adversarial reconstructions to ensure models are truly reasoning and not just memorizing.

Methods

The core of the authors' approach is prompt engineering applied to OpenAI's GPT-4 turbo model. They utilize a two-part prompting strategy:

  • Role Prompt: A system-level instruction that defines GPT-4's persona as a JSON-outputting assistant capable of solving riddles and brainteasers requiring complex reasoning. It specifies the desired output format, including the selected option's index and the answer text.
  • User Prompt: This prompt dynamically includes the specific brainteaser question and its corresponding answer options.

This method is applied uniformly to both sentence and word puzzles. The authors chose GPT-4 due to its advanced reasoning capabilities and large context window, which they believe contribute to its effectiveness on complex tasks.

Datasets

The primary dataset used is the BRAINTEASER dataset from the SemEval-2024 Task 9. This dataset consists of:

  • Sentence Puzzles: Brainteasers centered on sentence snippets that defy common sense.
  • Word Puzzles: Brainteasers that play on the default meaning of words, focusing on their letter composition.

The dataset includes original puzzles and adversarial reconstructions (Semantic Reconstruction and Context Reconstruction) designed to prevent memorization and test true reasoning. The test set is unlabeled, meaning the model must determine the correct answer from the provided options.

Key Insights and Ideas

  • Prompt Engineering is Key: The paper strongly suggests that carefully designed prompts can unlock sophisticated reasoning abilities in LLMs, even for tasks that go against typical common sense.
  • GPT-4's Adaptability: GPT-4, when guided by appropriate prompts, demonstrates a strong ability to tackle lateral thinking puzzles, outperforming other models like ChatGPT (zero-shot) and RoBERTa-L.
  • Adversarial Testing Matters: The inclusion of adversarial subsets highlights the importance of evaluating LLMs on their reasoning rather than their memorization capabilities, a crucial aspect for real-world problem-solving.

Key Results

The system achieved 9th place on the SemEval-2024 BRAINTEASER leaderboard.

  • Sentence Puzzle Results: GPT-4 + prompt engineering achieved an overall accuracy of 90.0%, significantly outperforming ChatGPT (62.68%) and RoBERTa-L (43.38%). It performed particularly well on the original puzzles (97.5%) and the O&S (Original & Semantic) subset (92.5%).
  • Word Puzzle Results: The system achieved an overall accuracy of 91.7%, again surpassing ChatGPT (53.46%) and RoBERTa-L (20.73%). It showed strong performance across all subsets, including Original (93.8%) and O&S (93.8%).

The paper notes that the human baseline was higher, indicating room for improvement, but the LLM's performance is highly competitive.

Limitations and Future Work

  • Human Baseline: The authors acknowledge that their system's performance, while strong, is still below the human baseline on both sentence and word puzzles. This suggests that human intuition and understanding of nuanced lateral thinking are still superior.
  • Future Work: The paper doesn't explicitly detail future work, but the results imply that further refinement of prompt engineering techniques, exploration of different LLM architectures, or hybrid approaches combining LLMs with symbolic reasoning could lead to even better performance. The success on this task also opens avenues for applying similar prompt engineering strategies to other creative or unconventional reasoning tasks.

Summary updated

BibTeX
@inproceedings{venkatesh2024bits,
  author = {Dilip Venkatesh and Yashvardhan Sharma},
  title = {{BITS Pilani at SemEval-2024 Task 9: Prompt Engineering with GPT-4 for Solving Brainteasers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure