TL;DR
This paper tackles the challenge of making Large Language Models (LLMs) better at "lateral thinking" puzzles, which require unconventional reasoning. The authors propose a novel prompting strategy combining static and dynamic few-shot examples with self-generated reasoning to significantly improve LLM performance on these tricky brain teasers.
What Problem Does This Paper Address?
Human cognition involves two modes of thinking: vertical (logical, analytical) and lateral (creative, divergent). While LLMs excel at vertical thinking, they often struggle with lateral thinking, which involves challenging assumptions and exploring novel perspectives. This paper focuses on SemEval-2024 Task 9: BRAINTEASER, a benchmark designed to evaluate LLMs' lateral thinking capabilities through sentence and word puzzles. These puzzles require models to go beyond default common-sense associations and default word meanings, making them particularly difficult for current AI systems. The challenge lies in developing prompting strategies that can elicit this unconventional, "outside-the-box" thinking from LLMs.
Methods
The researchers employed a multi-pronged approach to enhance LLM performance on the BRAINTEASER task, primarily using the Gemini 1.0 Pro model.
- Zero-Shot Prompting: Initially, they explored a basic zero-shot approach where the model was directly asked to choose the correct answer. They then refined this by providing the model with a definition of the puzzle type (sentence or word puzzle) before asking for the answer.
- Few-Shot Prompting: To improve performance further, they implemented few-shot learning. This involves providing the model with a few examples of correct question-answer pairs to guide its learning.
- Contextualized Example Selection (Dynamic Few-Shot): A key innovation is their dynamic few-shot approach. Instead of using a fixed set of examples, they use BERT-Large to encode test questions and training questions, then employ cosine similarity to select the most semantically similar training examples for each test question. This ensures the model learns from highly relevant in-context examples.
- Self-Generated Reasoning: Another significant contribution is the integration of self-generated reasoning. They prompt Gemini Pro 1 and GPT-4 to generate detailed explanations for why the correct answer is correct and why distractors are incorrect. This reasoning is then included in the prompt for the main task, aiming to deepen the model's understanding and improve its precision.
Datasets
The primary dataset used is from SemEval-2023 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. This dataset comprises two subtasks:
- Sentence Puzzles: These are multiple-choice questions centered around sentence snippets that require defying common-sense associations.
- Word Puzzles: These are multiple-choice questions where the answer violates the default meaning of a word, focusing instead on its letter composition.
The dataset was split into training and testing sets, with specific numbers provided in Table 1:
- Sentence Puzzle: 507 training samples, 120 test samples.
- Word Puzzle: 396 training samples, 96 test samples.
The paper also references baseline scores from Jiang et al. (2023b) for comparison, which were derived from Instruction, Common-sense, and Human categories.
Key Insights and Ideas
- Context is King (for Sentence Puzzles): For sentence puzzles, incorporating contextual information (like the definition of the puzzle type) significantly boosted performance, even in zero-shot settings.
- Reasoning Enhances Performance: Providing models with explanations (either human-generated or model-generated) for correct answers and distractors demonstrably improved accuracy. This suggests that explicit reasoning helps LLMs grasp the nuances of lateral thinking.
- Dynamic Examples Outperform Static (Sometimes): The dynamic, contextually relevant example selection showed promise, especially when combined with reasoning, suggesting that tailored in-context learning is more effective than a one-size-fits-all approach.
- Word Puzzles are Tricky: While improvements were seen, word puzzles proved particularly challenging. The paper notes that static examples and reasoning sometimes performed as well as or better than dynamic examples for word puzzles, indicating a different dynamic at play compared to sentence puzzles.
- LLMs Still Lag Humans: Despite significant improvements over baselines, the proposed methods still fell short of human annotator performance, highlighting the inherent difficulty of true lateral thinking for AI.
Key Results
The paper reports significant improvements over baseline models (Chat-GPT 0 Shot and Roberta-L) across both subtasks.
- Sentence Puzzles: The best-performing strategy, "1 Shot + SE + GPTR" (1-shot, Static Examples + GPT-4 Reasoning), achieved an overall accuracy of 0.775, a substantial leap from the baseline's 0.6268. Incorporating dynamic examples and reasoning also showed strong results, with "5 Shot + DE + Reason" reaching 0.758.
- Word Puzzles: The top strategy, "5 Shot + DE" (5-shot, Dynamic Examples), achieved an overall accuracy of 0.823, significantly outperforming the baseline of 0.5346. Other strong contenders included "5 Shot + DE + Reason" at 0.792.
The results tables (Table 2 and 3) detail performance across various strategies and evaluation metrics (Ori, Sem, Con, Ori & Sem, Ori & Sem & Con, Overall), consistently showing the superiority of the proposed few-shot and reasoning-enhanced methods over zero-shot and baseline approaches.
Limitations and Future Work
- Human Performance Gap: The most significant limitation acknowledged is that the proposed methods, while strong, do not yet reach human-level performance on these lateral thinking tasks. This indicates that further research is needed to bridge this gap.
- Dynamic Example Trade-offs: For sentence puzzles, while dynamic examples with reasoning showed promise, adding self-generated reasoning to dynamic examples did not always yield commensurate improvements and sometimes resulted in a trade-off, impacting performance on certain question types.
- Word Puzzle Dynamics: The paper notes that for word puzzles, static examples and reasoning sometimes performed better than dynamic examples, suggesting that the optimal strategy might differ between the two subtasks.
Future work could focus on developing more sophisticated reasoning mechanisms, exploring different methods for example selection, and investigating why the dynamic example approach doesn't consistently outperform static examples for word puzzles. Further research into the fundamental differences between sentence and word puzzles in the context of LLM reasoning is also warranted.
Summary updated