TL;DR
This paper explores how well large language models (LLMs) can solve abstract reasoning puzzles, specifically comparing state-of-the-art models like GPT-4 with a smaller, open-source model, Mistral-7B. They found that while GPT-4 excels with zero-shot prompting, fine-tuning Mistral-7B can achieve competitive results, suggesting a path to better performance with more accessible models.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has traditionally focused on tasks requiring objective reasoning. However, there's a growing interest in evaluating LLMs' capabilities in more abstract and creative thinking, particularly in solving puzzles that defy common sense. The SemEval-2024 Task 9: Brainteaser was designed to address this gap by presenting sentence and word puzzles that challenge typical reasoning. This paper aims to benchmark different approaches, specifically zero-shot prompting and instruction fine-tuning, on this novel dataset to understand how LLMs handle such creative reasoning tasks.
Methods
The researchers investigated two main approaches:
- Zero-Shot Prompting: This involves providing the LLM with a task description and the input, without any prior examples of the task. They experimented with three different prompt strategies for both GPT-4 and Mistral-7B. These prompts varied in how they presented the riddle, the answer choices, and the expected output format, including variations that tried to guide the model towards considering all options or implicitly handling a "None of the above" choice.
- Instruction-based Fine-tuning: For the smaller Mistral-7B model, they employed instruction fine-tuning. This method trains the model on examples formatted as instructions, defining the task. They used a sharded version of Mistral-7B and applied Low-Rank Adaptation (LoRA) for efficient fine-tuning, reducing the number of trainable parameters significantly.
Datasets
The primary dataset used in this study is the Brainteaser dataset introduced for SemEval-2024 Task 9. This dataset was created by crawling internet riddles and then generating variants through semantic rephrasing and GPT-4-assisted context reconstruction. This was done to mitigate potential memorization issues in LLMs and to better evaluate lateral thinking. The dataset is divided into two sub-tasks: Sentence Puzzles (SP) and Word Puzzles (WP), with a train-test split of 80:20.
Key Insights and Ideas
- GPT-4 Dominance in Zero-Shot: Even with sophisticated prompting, GPT-4 consistently outperformed Mistral-7B in zero-shot settings, highlighting its superior general reasoning capabilities.
- Prompt Engineering Matters: For Mistral-7B, tweaking the zero-shot prompts significantly improved performance, especially for word puzzles, demonstrating the importance of careful prompt design.
- Fine-tuning as a Viable Alternative: Instruction fine-tuning Mistral-7B yielded results comparable to, though slightly lower than, the best zero-shot approaches on GPT-4, particularly in the sentence puzzle task. This suggests that fine-tuning can bridge the performance gap for smaller models.
- The "None of the Above" Challenge: The dataset's label distribution showed a bias towards the first three answer choices, with the "None of the above" option being less frequent. This imbalance impacted evaluation metrics, and the fine-tuned Mistral-7B showed an improvement in F1 scores for this specific label, indicating a need for metrics beyond simple accuracy.
Key Results
- GPT-4: Achieved the highest overall performance across both Sentence Puzzles and Word Puzzles using zero-shot prompting (e.g., average F1 scores of 0.783 for SP and 0.906 for WP with prompt 2).
- Mistral-7B (Zero-Shot): With optimized prompts (P2 and P3), Mistral-7B achieved respectable zero-shot performance, with average F1 scores around 0.500 for SP and 0.625 for WP.
- Mistral-7B (Fine-tuned): The instruction fine-tuned Mistral-7B model achieved an average F1 score of 0.792 for Sentence Puzzles and 0.844 for Word Puzzles, making it the second-best performing model overall and outperforming its own zero-shot performance.
Limitations and Future Work
The authors acknowledge that while they explored various prompts, there's a possibility that more effective prompts exist. Future work could involve automating prompt search and exploring additional datasets that focus on lateral thinking. They also suggest investigating other fine-tuning techniques to further enhance the performance of smaller models like Mistral-7B. The study also highlights the need for more nuanced evaluation metrics that can account for label imbalances in datasets.
Summary updated