TL;DR
This paper investigates whether Large Language Models (LLMs) can exhibit "lateral thinking" – the ability to think outside the box and solve problems creatively, akin to human intuition. The authors explore various prompting strategies and in-context learning techniques to enhance LLM performance on a lateral thinking benchmark, finding that concise prompts and dynamic sample selection significantly improve results.
What Problem Does This Paper Address?
Human cognition involves both logical (vertical) and creative (lateral) thinking. While LLMs have shown impressive capabilities in logical reasoning, their ability to perform lateral thinking – essential for tasks requiring creativity and challenging conventional assumptions – remains less understood. Existing benchmarks often focus on vertical reasoning, and there's a lack of thorough analysis on how different prompting methods impact LLMs' lateral thinking abilities. This paper aims to bridge this gap by evaluating and enhancing LLMs' performance on a dedicated lateral thinking task.
Methods
The researchers employed three main strategies to evaluate and improve LLMs' lateral thinking:
- Chain-of-Thought (CoT) Prompting: They explored variations of CoT, including "Simple-Internal-CoT" (where the LLM autonomously follows steps) and "External-CoT" (where the model is guided step-by-step externally).
- Enhanced Prompting Strategies: This involved providing LLMs with task descriptions, ranging from simple to detailed, and also a compressed version that retained essential information. They hypothesized that concise, informative prompts would be more effective.
- In-Context Learning with Retrieval Augmented Generation (RAG): To overcome the limitations of static few-shot examples, they developed a RAG pipeline to dynamically select relevant examples from the training data. This involved different RAG variants (Ordinary RAG, Ranked RAG, and RAG-Fusion) to find the most suitable examples for the LLM. They also experimented with including explanations and summaries of these examples.
Datasets
- BrainTeaser Dataset: This is the primary dataset used, specifically the Sentence Puzzle sub-task. It's a multiple-choice question-answering dataset designed to evaluate lateral thinking and challenge default commonsense associations. It was created by Jiang et al. (2023c) and contains 1,100 puzzles.
- Additional Datasets: SWAG (Zellers et al., 2018) and CommonsenseQA (Talmor et al., 2019) were used to evaluate the impact of fine-tuning on general commonsense reasoning.
Key Insights and Ideas
- Prompt Length Matters: Contrary to what one might expect, longer, more detailed prompts didn't always lead to better performance. Concise, informative prompts, especially the "compressed" versions, often yielded superior results, suggesting LLMs might favor brevity.
- Dynamic In-Context Learning is Crucial: Static few-shot examples are less effective for lateral thinking tasks. Dynamically selecting examples using RAG pipelines significantly improved performance, indicating that tailoring examples to the specific question is key.
- Explanations Can Hinder: Interestingly, excluding explanations from the RAG pipeline and allowing the LLM to infer the thinking path independently sometimes boosted performance. This suggests LLMs might have an inherent ability to extract relations, and explicit explanations might not always be necessary or could even introduce bias.
- Fine-tuning for Lateral Thinking Pays Off: Fine-tuning an LLM (Zephyr-7B-ß) on a dataset specifically designed to teach lateral thinking improved its performance on general commonsense reasoning tasks, challenging the notion that such specialized training might not generalize.
Key Results
- The paper's submission to SemEval-2024 Task 9 achieved a remarkable score of 0.975 on the Sentence Puzzle sub-task, significantly surpassing the baseline of 0.608 and securing the second-highest score.
- GPT-4, with a "Simple-Internal-CoT" and "Detailed" task description, achieved the highest score of 97.5% on the Sentence Puzzle sub-task.
- For GPT-3.5, the best performance was achieved with "Direct" prompting and a "Detailed" task description, reaching 82.5%.
- Fine-tuning Zephyr-7B-ß on the lateral thinking dataset improved its accuracy on SWAG from 38% to 46% and on CommonsenseQA from 31.33% to 36%.
Limitations and Future Work
- Model Dependence: The performance varied significantly across different LLMs (GPT-3.5, GPT-4, Zephyr-7B-ß), indicating that not all LLMs possess strong lateral thinking capabilities, and larger models with more data tend to perform better.
- Prompt Engineering Complexity: While they found effective prompting strategies, the optimal approach can be task-dependent, and further research is needed to fully understand the nuances of prompt engineering for lateral thinking.
- Computational Cost: The RAG-Fusion method, while effective, is noted as being time-consuming due to the multiple LLM inferences required.
- Future Work: The authors suggest further exploring the integration of out-of-the-box thinking into model training and investigating more efficient methods for dynamic in-context learning. They also highlight the potential for fine-tuning on lateral thinking datasets to enhance general reasoning abilities.
Summary updated