OUNLP at SemEval-2024 Task 9: Retrieval-Augmented Generation for Solving Brain Teasers with LLMs

Vineet Saravanan, Steven WilsonNAACL 2024

TL;DR

This paper explores how well Large Language Models (LLMs) like GPT-4 and Gemini can solve brain teasers, a task requiring both linguistic understanding and logical reasoning. They found that "one-shot" prompting, where a similar example is provided, significantly boosts performance, especially for sentence-based puzzles, highlighting the power of context for LLMs.

What Problem Does This Paper Address?

The paper tackles the challenge of evaluating and improving the reasoning capabilities of LLMs, particularly in tasks that mimic complex human cognition. Brain teasers, while seemingly simple, embed layers of semantics, pragmatics, and world knowledge that are notoriously difficult for AI to grasp. The authors aim to benchmark current LLMs on these puzzles and investigate how different prompting strategies, including retrieval-augmented generation (RAG) through "one-shot" examples, can enhance their performance. This research is crucial for understanding the current limits of AI reasoning and guiding future advancements beyond mere pattern recognition.

Methods

The core methodology revolves around prompt engineering to elicit better performance from LLMs on brain teasers. The researchers experimented with various prompt structures:

  • Basic Prompt: A straightforward instruction to pick the best choice.
  • Task Description Prompt: Included a description of the puzzle type (e.g., "sentence puzzle").
  • Chain-of-Thought (CoT) Prompting: Encouraged models to articulate their reasoning steps.
  • Explanation Prompting: Asked models to explain their rationale.
  • One-Shot In-Context Learning: This was a key focus. They used a large dataset of riddles to find either a similar or dissimilar example to present to the LLM alongside the actual brain teaser. This was implemented using sentence embeddings (all-MiniLM-L6-v2) to calculate cosine similarity for selecting examples.
  • Council Method: A novel approach simulating a "council" of three hypothetical individuals discussing a puzzle to reach a consensus.

Datasets

  • Brainteaser Dataset: The authors web-scraped example riddles for their experiments. The SemEval-2024 Task 9: BRAINTEASER dataset was used for evaluation.
  • Riddle Dataset: A collection of 3,899 riddles was extracted from riddles.com for use in the one-shot learning approach.

Key Insights and Ideas

  • One-Shot Learning is King: Providing a single, relevant example (one-shot) dramatically improved LLM accuracy, especially for sentence puzzles. This suggests LLMs are highly sensitive to contextual priming.
  • Similarity Matters (Sometimes): While both similar and dissimilar examples were tested, the paper found that the similar one-shot approach often yielded better results for word puzzles, while the dissimilar one was better for sentence puzzles. This hints at nuanced ways context can guide reasoning.
  • Task Type Nuance: The Word Puzzle task proved significantly harder than the Sentence Puzzle task, suggesting that puzzles requiring deeper semantic and cultural understanding pose a greater challenge for current LLMs.
  • CoT and Explanation Don't Always Boost Accuracy: While these methods enhance interpretability, they didn't consistently lead to higher accuracy in solving brain teasers, suggesting that the underlying reasoning process might not always benefit from explicit step-by-step articulation.
  • Proprietary vs. Open-Source Gap: Larger, proprietary models like GPT-4 and Gemini significantly outperformed open-source models, highlighting the impact of resources, data, and proprietary algorithms.

Key Results

  • Top Performance: The one-shot prompting method consistently achieved top accuracy scores across the tested models.
  • Model Performance: GPT-4 and Gemini generally outperformed GPT-3.5. Gemini showed remarkable accuracy even with a basic prompt on sentence puzzles, indicating strong inherent capabilities.
  • Specific Prompt Success: The best submission achieved an accuracy of 0.925 for sentence puzzles and 0.9375 for word puzzles using a GPT-4 one-shot different prompt for sentence puzzles and a GPT-4 one-shot similar prompt for word puzzles.
  • Ranking: The second submission ranked 14th overall in the SemEval-2024 BRAINTEASER task.
  • Word Puzzle Difficulty: Accuracy rates were generally lower for word puzzles compared to sentence puzzles across all models.

Limitations and Future Work

  • Generalization vs. Specialization: The paper notes that GPT-4's broad training might lead to it not performing as well on highly specialized tasks like brain teasers compared to models with more focused training.
  • Council Method Ineffectiveness: The "council" method, while an interesting attempt to mimic human collaboration, did not yield significant improvements in accuracy for word puzzles.
  • Open-Source Model Gap: The substantial performance difference between proprietary and open-source models indicates a need for further development in the latter.
  • Future Research: The authors suggest further research into why CoT and explanation strategies don't always improve accuracy, and exploring how to bridge the gap in performance for word puzzles, which may require deeper cultural and semantic understanding.

Summary updated

BibTeX
@inproceedings{saravanan2024ounlp,
  author = {Vineet Saravanan and Steven Wilson},
  title = {{OUNLP at SemEval-2024 Task 9: Retrieval-Augmented Generation for Solving Brain Teasers with LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure