Abstract
Large Language Models (LLMs) have demonstrated remarkable proficiency in "vertical thinking"—step-by-step logical reasoning. However, "lateral thinking," which involves solving puzzles by breaking conventional patterns and challenging default assumptions, remains a significant hurdle. This paper investigates strategies to unlock lateral reasoning in LLMs through optimized prompting, a novel RAG-Fusion pipeline for dynamic in-context learning, and fine-tuning on a specialized "thinking paths" dataset. Our approach achieved a 97.5% accuracy on the BrainTeaser benchmark, ranking second in the SemEval-2024 competition. Furthermore, we demonstrate that lateral thinking training generalizes to and improves performance on standard commonsense reasoning tasks.
Introduction: The Challenge of Creative Reasoning
Most AI evaluation frameworks prioritize vertical thinking—the linear, logical progression used in mathematics or instruction following. Human intelligence, conversely, is characterized by the ability to think laterally. Lateral thinking is essential for solving riddles, where the solution often lies in identifying a hidden assumption or shifting perspective.
LLMs often struggle with these tasks because their training corpora are dominated by straightforward, logical discourse. When faced with a riddle like "A man shaves every day, yet keeps his beard long," a model might fail by assuming the man shaves himself. The goal of this research is to bridge this gap by exploring how sophisticated prompting and retrieval mechanisms can activate creative reasoning capabilities.
Methodology
The researchers developed a multi-faceted framework to enhance lateral reasoning performance.
1. Task Informative Contexts
The team evaluated three levels of task description to prime the model:
- Simple: A basic definition of the task.
- Detailed: An exhaustive prompt explaining 11 common "tricks" found in riddles (e.g., social biases and default assumptions).
- Compressed: A distilled version of the detailed prompt designed to provide essential guidance without overwhelming the model's context window.
2. Chain of Thought (CoT) Variations
Two primary CoT strategies were employed:
- Internal CoT: The model is prompted to "think step-by-step" internally before providing an answer.
- External CoT: A "thesis-based" approach where the model generates a potential logical path (a "thesis") connecting the riddle to each possible answer choice. It then evaluates these paths to select the most plausible one.
3. RAG-Fusion for Dynamic In-Context Learning
Static few-shot prompting often fails to capture the specific nuances of a riddle. To address this, the authors proposed a "RAG-Fusion" pipeline.
In this pipeline, the LLM generates four variations of the input riddle: semantic reconstructions (similar meaning) and contextual reconstructions (similar situation). These variations are used to query a vector store for relevant examples from the training set. The retrieved "shots" are then ranked and integrated into the final prompt.
4. Lateral Thinking Tuning
To further bake creative reasoning into the model, the researchers used GPT-4 to generate "thinking paths" for a large set of riddles. These paths delineate the logic required to reach the correct answer. The Zephyr-7B-β model was then fine-tuned on this custom dataset.
Datasets and Evaluation
The primary evaluation was conducted on the BrainTeaser benchmark, which includes Sentence Puzzles (627 samples) and Word Puzzles (492 samples). To test generalization, the models were also evaluated on SWAG and CommonsenseQA (CSQA), which are standard benchmarks for non-riddle commonsense reasoning.
Results and Analysis
The experimental results highlighted several critical factors in model performance, particularly the interaction between prompt detail and reasoning style.
As shown in the figure, "Compressed" prompts often outperformed "Detailed" ones in specific settings like External CoT. This suggests a "Goldilocks" effect where providing too much information can clutter the context and degrade performance. GPT-4 consistently outperformed smaller models, indicating that lateral thinking may be an emergent property of scale.
Performance Gains and Generalization
- Top Accuracy: The best configuration (GPT-4 with Simple Internal-CoT and Detailed task description) reached 97.5% accuracy on Sentence Puzzles.
- Transfer Learning: Fine-tuning Zephyr-7B-β on lateral thinking paths improved its SWAG score from 38% to 46% and its CSQA score from 31.33% to 36%. This confirms that learning to "think outside the box" enhances general language understanding.
- Retrieval Efficiency: The RAG-Fusion method achieved a hit rate of 0.767, outperforming standard RAG (0.65).
Discussion and Conclusion
The study demonstrates that while LLMs are primarily trained on logical text, they possess a latent capacity for lateral thinking that can be unlocked with the right tools. Dynamic retrieval via RAG-Fusion and the generation of explicit "thinking paths" are highly effective strategies. Future work should focus on reducing the computational overhead of RAG-Fusion and exploring how lateral thinking can be integrated into the core pre-training phase of future models.
Summary updated