BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense

Baktash Ansari, Mohammadmostafa Rostamkhani, Sauleh EetemadiNAACL 2024

TL;DR

This paper tackles the challenge of evaluating "lateral thinking" in language models, moving beyond traditional logical reasoning. They introduce the BRAINTEASER task and explore fine-tuning BERT/RoBERTa and using Chain-of-Thought prompting with large language models (LLMs), culminating in a "ReConcile" method for collaborative model consensus.

What Problem Does This Paper Address?

The NLP community has largely focused on "vertical thinking" – sequential, logical, and rule-based reasoning. However, many real-world problems require "lateral thinking," a more divergent and creative process of exploring new perspectives. SemEval-2024 Task 9, BRAINTEASER, aims to bridge this gap by evaluating LLMs' ability to think outside the box. The task presents multi-choice questions designed to challenge common sense reasoning and encourage creative problem-solving, moving beyond rote memorization or purely analytical approaches.

Methods

The authors employed a multi-pronged approach:

  • Fine-tuning: They fine-tuned two established transformer models, BERT and RoBERTa-Large, on the BRAINTEASER dataset. This involved concatenating questions with choices and feeding them through the models.
  • Zero-Shot Prompting with Chain-of-Thought (CoT): They leveraged six large language models (LLMs) like GPT-3.5, Mixtral, and Llama2, using CoT prompting. This technique encourages the LLM to generate step-by-step reasoning before providing an answer, aiming to improve accuracy and reduce errors.
  • ReConcile: This novel technique, adapted from previous work, involves a "round table conference" of multiple LLMs. It proceeds in three phases:
  • Initial Response Generation: Each model provides an answer, reasoning, and a confidence score.
  • Multi-Round Discussion: Models receive the outputs of others and refine their answers.
  • Final Answer Generation: A consensus is reached by weighting the models' confidence scores. This method was applied to three LLMs (Mixtral, GPT3.5, Claude) to foster collaborative decision-making.

Datasets

The primary dataset used is BRAINTEASER, specifically the English version for SemEval-2024 Task 9. This dataset consists of two subtasks:

  • Sentence Puzzles: Brain teasers focused on sentence snippets where common sense is defied.
  • Word Puzzles: Brain teasers where the answer relies on the letter composition of a word, deviating from its default meaning.

The dataset is split into training and testing sets, with both semantic and context variants to ensure lateral thinking is evaluated rather than memorization. For fine-tuning, both sentence and word puzzles were used, while zero-shot prompting primarily utilized the sentence puzzles test set.

Key Insights and Ideas

  • Lateral thinking is a distinct challenge: The BRAINTEASER task highlights that current LLMs, even powerful ones, struggle with creative, non-linear problem-solving that deviates from standard logical deduction.
  • CoT improves LLM reasoning: The Chain-of-Thought prompting significantly boosted LLM performance by forcing them to articulate their reasoning process, making them less prone to simple errors.
  • Collaboration enhances performance: The ReConcile method, where multiple LLMs discuss and reach a consensus, demonstrably improved accuracy over individual model performance. This suggests that a "society of minds" approach can be effective for complex, ambiguous tasks.
  • Microsoft Copilot excels at "tricky" questions: Notably, Microsoft Copilot achieved the highest zero-shot performance, indicating a strong ability to recognize and handle questions that play on words or require understanding subtle nuances, rather than just logical deduction.

Key Results

  • Fine-tuned models: The best fine-tuned model, RoBERTa-Large, achieved an overall accuracy of 0.766 on sentence puzzles and 0.645 on word puzzles.
  • Zero-shot LLMs: Microsoft Copilot achieved the highest zero-shot performance on sentence puzzles with an accuracy of 0.860.
  • ReConcile: The ReConcile method, after two discussion rounds, achieved a consensus accuracy of 0.758 on sentence puzzles, showing improvement over individual model initial rounds. Claude consistently showed high individual performance within the ReConcile framework.

Limitations and Future Work

  • Model performance disparity: The authors note that the ReConcile method works best when models have similar performance levels. A model with significantly lower performance could negatively impact the consensus.
  • Calculation accuracy: Microsoft Copilot, despite its strong performance, sometimes struggles with calculation-based questions, occasionally misinterpreting options or attempting to force a fit when the correct answer isn't present.
  • Future work: The paper suggests that the ReConcile iterative process can be continued until full consensus is reached. Further exploration into how to best balance diverse model strengths and weaknesses within collaborative frameworks is implied. The success of Microsoft Copilot also points to the importance of models that can effectively identify and leverage wordplay and non-literal interpretations.

Summary updated

BibTeX
@inproceedings{ansari2024bamo,
  author = {Baktash Ansari and Mohammadmostafa Rostamkhani and Sauleh Eetemadi},
  title = {{BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure