TL;DR
This paper introduces an iterative system for optimizing Chain-of-Thought (CoT) prompting for large language models (LLMs) on lateral thinking puzzles. By systematically evaluating model outputs and incorporating human feedback, they significantly improve LLM performance on challenging, adversarial datasets, offering a novel approach to prompt engineering and dataset analysis.
What Problem Does This Paper Address?
While LLMs excel at logical reasoning tasks, their ability to tackle "lateral thinking" puzzles – those requiring creative, "outside-the-box" problem-solving – remains less explored. These puzzles, often found in datasets like BRAINTEASER, are designed to prevent memorization through adversarial construction, making them difficult for standard LLM approaches. The authors aim to improve LLM performance on these lateral thinking tasks by developing a more sophisticated prompt engineering strategy.
Methods
The core of the paper is an iterative system for optimizing CoT prompting. The process involves:
- Naive CoT Prompting: Initially, they generate simple CoT prompts by sampling training data and observing LLM outputs. This helps identify common reasoning patterns and potential failure points.
- Human Evaluation: They conduct human evaluations on different subsets of the data (Base, SR, CR) to identify specific challenges and errors that humans and models struggle with. This step is crucial for establishing a human benchmark.
- Iterative Prompt Refinement: Based on the human evaluation insights, they iteratively refine the CoT prompts. This includes developing prompts that encourage the model to reason over all answer choices, explaining why incorrect ones are wrong, not just why the correct one is right. They also create "CoT-Mix" prompts weighted towards more challenging adversarial datasets (SR and CR).
- Data Analysis: The system also provides insights into the dataset itself, helping to identify problematic questions and inform future data collection and synthesis.
Datasets
The primary dataset used is BRAINTEASER, specifically the sentence puzzle subtask from SemEval-2024 Task 9. This dataset includes:
- Base Set: Original English-language riddles and brainteasers.
- SR (Semantic Reconstruction) Set: Rephrased versions of the original questions, maintaining the same answer choices.
- CR (Context Reconstruction) Set: Questions with altered situational contexts but the same misleading premise.
These SR and CR sets are designed to be adversarial and prevent memorization. The authors also generated their own synthetic datasets for evaluation.
Key Insights and Ideas
- Lateral Thinking is a Distinct Challenge: LLMs struggle with lateral thinking puzzles, which require more than just sequential logical inference.
- Human Evaluation is Key for Prompt Engineering: Incorporating human performance as a benchmark is vital for identifying subtle reasoning failures and guiding prompt optimization.
- Disproving Incorrect Answers Matters: Prompting LLMs to explain why incorrect answers are wrong, not just why the correct one is right, significantly improves performance and reduces reliance on memorization.
- Adversarial Datasets Reveal Deeper Issues: The SR and CR datasets, designed to be challenging, highlight specific weaknesses in LLM reasoning that might not be apparent in simpler datasets.
- Dataset Quality is Crucial: The iterative evaluation process can identify problematic or ambiguous questions within the dataset, informing future data curation and generation efforts.
Key Results
The proposed iterative CoT prompting system led to significant improvements:
- Accuracy Gains: The "New CoT-Mix" prompts achieved the highest accuracy across various metrics, outperforming naive CoT prompts and even approaching human consensus on some subsets. For instance, the "GPT-4 New CoT-Mix" achieved 95.0% accuracy on the Base set and 92.5% on the SR set, and 77.5% on the Adversarial set, showing substantial gains over "GPT-4 Zero Shot" (76.7% overall).
- Improved Robustness: The gains on adversarial datasets (SR, CR, Adversarial) suggest the system helps LLMs rely less on memorization and more on genuine reasoning.
- Competitive Performance: Their official submission using "GPT-4 New CoT-Base" prompts scored 2nd overall on the base data and achieved competitive rankings on other metrics in the SemEval-2024 Task 9.
Limitations and Future Work
- Human Evaluation Bottleneck: While effective, human evaluation can be time-consuming and may introduce annotator bias.
- Dataset Specificity: The current system is optimized for the BRAINTEASER dataset. Adapting it to other types of open-ended reasoning tasks might require further work.
- Future Directions: The authors plan to implement techniques for creating synthetic data and applying their evaluation framework to a broader, open-ended dataset. They also aim to explore different prompt engineering techniques.
Summary updated