ROSHA at SemEval-2024 Task 9: BRAINTEASER A Novel Task Defying Common Sense

Mohammadmostafa Rostamkhani, Shayan Mousavinia, Sauleh EetemadiNAACL 2024

TL;DR

This paper tackles the "BRAINTEASER" task, which challenges language models to solve riddles requiring lateral thinking and common sense beyond literal interpretation. The authors explore various fine-tuning and prompting strategies, including a novel "round-table" approach with multiple LLMs, achieving strong results, particularly with GPT-4.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has made significant strides in tasks requiring logical and sequential reasoning. However, a notable gap exists in models' ability to handle "lateral thinking" puzzles. These riddles often rely on understanding implicit meanings, challenging assumptions, and employing common sense that deviates from straightforward interpretations. The BRAINTEASER task, introduced in SemEval-2024, aims to bridge this gap by evaluating models on their capacity to solve sentence and word puzzles that defy common sense associations.

Methods

The researchers employed a multi-pronged approach:

  • XLM-RoBERTa Fine-tuning: They fine-tuned the XLM-RoBERTa model on the original BRAINTEASER dataset, as well as augmented versions including the BiRdQA and RiddleSense datasets.
  • Word Extender Strategy: For word puzzles, they innovatively expanded each word option into a full sentence using ChatGPT-3.5. This aimed to provide richer semantic context for the model during training.
  • RECONCILE (Round-Table Conference): They adapted the RECONCILE method, which involves multiple LLMs collaborating. They used ChatGPT, Gemini, and Mixtral-8x7B in a four-round iterative process where models prompt each other, providing confidence scores and rationales. This method did not involve fine-tuning.
  • GPT-4 Zero-Shot: They also evaluated the zero-shot performance of GPT-4 using a Copilot GUI, prompting it to think step-by-step.

Datasets

  • Original BRAINTEASER Dataset: The primary dataset for the task, comprising sentence and word puzzles.
  • BiRdQA Dataset: A bilingual dataset for question answering on challenging riddles.
  • RiddleSense Dataset: A dataset focused on riddles requiring linguistic creativity and common sense.

The authors also describe preprocessing steps to adapt these datasets, such as ensuring a consistent number of options per riddle.

Key Insights and Ideas

  • Lateral Thinking is Hard: The BRAINTEASER task highlights that current LLMs struggle with reasoning that goes beyond literal interpretation and requires common sense that deviates from typical associations.
  • Data Augmentation Helps: Incorporating external riddle datasets like BiRdQA and RiddleSense significantly improved the performance of the XLM-RoBERTa model.
  • Sentence Expansion for Word Puzzles: Expanding individual word options into full sentences proved to be an effective strategy for improving word puzzle performance, suggesting that richer context aids comprehension.
  • Collaborative Reasoning (Round-Table) Shows Promise: The RECONCILE approach, where multiple LLMs iteratively refine answers and provide reasoning, demonstrated strong performance, particularly for sentence puzzles.
  • Gemini's Consistent Performance: Gemini consistently outperformed other models in the round-table experiments across multiple rounds.
  • GPT-4's Superiority: GPT-4, even in a zero-shot setting, significantly outperformed all other models and methods tested, underscoring its advanced reasoning capabilities.
  • Round-Table Nuances: While beneficial for sentence puzzles, the round-table approach showed a performance decrease for word puzzles, possibly due to cascading incorrect reasoning. The optimal number of rounds varied between sentence (3) and word (1) puzzles.

Key Results

  • The fine-tuned XLM-RoBERTa model, when augmented with BiRdQA and RiddleSense, achieved scores of 0.900 for sentence puzzles (S_ori) and 0.906 for word puzzles (W_ori), representing their most successful experiment.
  • In the round-table experiments, Gemini consistently achieved high scores, often exceeding 0.750 for sentence puzzles.
  • GPT-4 demonstrated exceptional performance, achieving 0.900 for S_ori and 0.906 for W_ori in its zero-shot evaluation, surpassing all other models.

Limitations and Future Work

  • Round-Table Discrepancy: The paper notes a performance decrease in word puzzles with the round-table approach, suggesting further investigation into why this occurs and how to mitigate it.
  • Optimal Rounds: The study suggests that 3 rounds are sufficient for sentence puzzles, while 1 round is preferable for word puzzles, indicating a need for task-specific optimization of the round-table process.
  • GPT-4's Dominance: The significant superiority of GPT-4 suggests that future work should focus on understanding and replicating its advanced reasoning capabilities, potentially through more sophisticated prompting or architectural innovations.
  • Further Exploration of LLM Collaboration: The success of the round-table approach warrants further research into optimal configurations and strategies for multi-LLM collaboration on complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{rostamkhani2024rosha,
  author = {Mohammadmostafa Rostamkhani and Shayan Mousavinia and Sauleh Eetemadi},
  title = {{ROSHA at SemEval-2024 Task 9: BRAINTEASER A Novel Task Defying Common Sense}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure