IIMAS at SemEval-2024 Task 9: A Comparative Approach for Brainteaser Solutions

Cecilia Reyes, Orlando Ramos-flores, Diego Martínez-maquedaNAACL 2024

TL;DR

This paper presents a comparative study of transformer-based models for solving "brainteasers," a novel task requiring common-sense reasoning and unconventional thinking. The IIMAS team fine-tuned BERT and RoBERTa models, augmenting the dataset with LLM-generated examples, achieving a best accuracy of 0.93. While successful, the study highlights the persistent challenge of word puzzles, suggesting a need for deeper contextual understanding.

What Problem Does This Paper Address?

The SemEval-2024 Task 9, "BRAINTEASER: A Novel Task Defying Common Sense," introduces a unique challenge: solving riddles and puzzles that require "out-of-the-box" thinking, often defying literal interpretations. These brainteasers are categorized into Sentence Puzzles (defying common sense within sentence fragments) and Word Puzzles (challenging predefined word meanings through letter composition). Traditional language models excel at "vertical" or convergent thinking but struggle with the "lateral" or divergent thinking needed to crack these puzzles. This task aims to push the boundaries of natural language understanding by evaluating models' ability to engage in this unconventional reasoning.

Methods

The IIMAS team employed a classification framework for tackling brainteasers. Their core strategy involved fine-tuning pre-trained transformer models. Initially, they experimented with a zero-shot approach using the BART model on a MultiNLI dataset, which yielded poor results. This led them to fine-tune models on the provided brainteaser dataset.

Specifically, they fine-tuned:

  • BERT (bert-base-uncased): Data was transformed into premise-hypothesis pairs, with labels "Entailment" (correct answer), "Contradiction" (distractor1/distractor2), and "Neutral" (distractor-unsure).
  • RoBERTa Winogrande: This model was adapted to predict a boolean value (True/False) based on the relationship between the premise and hypothesis.

To enhance performance, they augmented the original dataset with synthetic examples generated by large language models (LLMs) like ChatGPT and Gemini. These LLM-generated examples underwent manual review to ensure accuracy.

Datasets

  • Original SemEval-2024 Task 9 Dataset: Comprised 507 sentence puzzles and 396 word puzzles, each with one correct answer and three distractors.
  • Augmented Dataset: The original dataset was expanded with synthetic examples generated by ChatGPT and Gemini, resulting in a total of 4,644 labeled pairs for fine-tuning.

Key Insights and Ideas

  • LLM Augmentation is Crucial: Generating additional data with LLMs like ChatGPT and Gemini significantly boosted performance, demonstrating their utility in creating diverse and challenging training examples.
  • BERT Outperforms RoBERTa: The BERT model, after fine-tuning, showed superior performance compared to the RoBERTa Winogrande model for this specific task.
  • Word Puzzles Remain a Bottleneck: Despite extensive efforts, word puzzles proved significantly more challenging than sentence puzzles. This suggests that current models, even with fine-tuning, struggle with the nuanced, often pun-based or letter-manipulating nature of word puzzles, requiring a deeper, more figurative understanding.
  • Data Pruning is Necessary: The initial inclusion of all synthetically generated data, particularly from word puzzles, led to suboptimal performance. Pruning the dataset, specifically discarding word puzzle data generated by LLMs, improved the BERT model's metrics for sentence puzzles.

Key Results

The IIMAS team achieved a best accuracy of 0.93 with their fine-tuned BERT model (Model 3, after data pruning). They also reported an F1 score of 0.87 for the Entailment class, 0.94 for the Contradiction class, and 0.96 for the Neutral class. Their overall ranking in the SemEval-2024 Task 9 was 33rd out of 50 participants, with an average score of 0.459 (0.658 for Sentence Puzzle, 0.260 for Word Puzzle).

Limitations and Future Work

The authors acknowledge that the primary errors are associated with the word puzzle task, attributing this to class imbalance and the inherent difficulty of requiring deeper contextual understanding and a more figurative sense. They suggest that future work should focus on addressing this challenge, potentially through more advanced techniques for contextual understanding and figurative language processing. The paper also notes that while their results surpassed the baseline, there is still significant room for improvement, especially in the resolution of word puzzles.

Summary updated

BibTeX
@inproceedings{reyes2024iimas,
  author = {Cecilia Reyes and Orlando Ramos-flores and Diego Martínez-maqueda},
  title = {{IIMAS at SemEval-2024 Task 9: A Comparative Approach for Brainteaser Solutions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure