TL;DR
This paper tackles the challenge of evaluating large language models (LLMs) on "lateral thinking" puzzles, which require going beyond common-sense assumptions. The authors applied a zero-shot DeBERTa model to the BRAINTEASER benchmark, achieving a modest score but highlighting the difficulty of this task for current models.
What Problem Does This Paper Address?
Human reasoning can be broadly categorized into "vertical thinking" (logical, sequential) and "lateral thinking" (creative, unconventional). While NLP models, particularly LLMs, have shown impressive capabilities in vertical thinking tasks, their ability to handle lateral thinking puzzles remains largely unexplored. This is partly because datasets often filter out puzzles that deviate from common-sense associations. To address this gap, the SemEval-2024 Task 9 introduced the BRAINTEASER benchmark, specifically designed to assess LLMs' lateral thinking skills through multiple-choice question-answering puzzles. These puzzles require models to move beyond literal interpretations and common-sense defaults to find the correct solution.
Methods
The authors employed a zero-shot learning approach using the mDeBERTa V3 base model. DeBERTa is a Transformer-based architecture known for its improvements over BERT and RoBERTa, including disentangled attention mechanisms and an enhanced mask decoder. The multilingual version (mDeBERTa) was chosen for its ability to handle diverse linguistic data. The zero-shot strategy meant the model was not explicitly fine-tuned on the BRAINTEASER dataset. Instead, the researchers focused on prompt engineering, crafting prompts that framed the questions as brain-teasers and included the context, question, and answer choices. They experimented with various prompt strategies to elicit the best responses.
Datasets
The primary dataset used is BRAINTEASER, introduced for SemEval-2024 Task 9. This benchmark consists of two subtasks: Sentence Puzzles and Word Puzzles. The paper focuses on the Sentence Puzzle subtask. The dataset was curated from publicly available websites and includes original puzzles along with semantically and contextually reconstructed versions to challenge models further and mitigate memorization. The authors did not perform additional fine-tuning on any datasets.
Key Insights and Ideas
- Lateral thinking is a distinct challenge for LLMs: The paper underscores that even powerful LLMs struggle with puzzles that require deviating from common-sense assumptions, suggesting a fundamental difference between pattern recognition and true creative reasoning.
- Zero-shot DeBERTa shows promise but limitations: While DeBERTa is a strong baseline, its performance on this task, even with careful prompt engineering, indicates that current zero-shot capabilities are insufficient for complex lateral thinking.
- Prompt engineering is crucial but not a silver bullet: The authors found that framing the task as a "brain-teaser" and carefully structuring the input prompt was essential for eliciting any relevant responses, but it didn't overcome the core reasoning deficit.
- Adversarial reconstructions are effective: The inclusion of semantic and contextually reconstructed puzzles in BRAINTEASER is a valuable addition for evaluating model robustness beyond simple memorization.
Key Results
The proposed DeBERTa model achieved an aggregate score of 0.250 on the BRAINTEASER Sentence Puzzle task. This score is presented as a starting point, indicating significant room for improvement. The paper also includes a table comparing their results to the top-performing teams, where their straightforward zero-shot approach was outperformed by models that likely employed more sophisticated techniques or fine-tuning.
Limitations and Future Work
The authors acknowledge that their approach achieved a modest score, suggesting that the task is indeed challenging for current LLMs. They explicitly state that their model achieved an aggregate score of 0.250, indicating "a significant room for improvements in future works."
Future work directions suggested include:
- Leveraging few-shot learning capabilities of LLMs.
- Exploring alternative models like GPT and T5.
- Integrating additional data or samples from training and development sets.
- Fine-tuning the model and framing the problem as a text classification task.
Summary updated