ALF at SemEval-2024 Task 9: Exploring Lateral Thinking Capabilities of LMs through Multi-task Fine-tuning

Seyed Ali Farokh, Hossein ZeinaliNAACL 2024

TL;DR

This paper explores how well large language models (LLMs) can handle "lateral thinking" puzzles, which require creative reasoning beyond straightforward logic. By fine-tuning BERT and T5-based models on the Brain Teaser benchmark, the authors found that DeBERTa-v3-large achieved impressive results, even surpassing human performance on some metrics, highlighting the potential of these models for complex reasoning tasks.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen rapid advancements, leading to the development of sophisticated benchmarks for evaluating model capabilities. However, many existing benchmarks focus on commonsense reasoning or linear inference. The SemEval 2024 Task 9, "Brain Teaser," introduces a unique challenge: evaluating a model's ability to perform "lateral thinking." This type of reasoning often involves looking at problems from unconventional perspectives, defying default assumptions, and employing creative problem-solving. Unlike benchmarks that rely on straightforward associations, Brain Teaser puzzles require models to go beyond simple logical chains and engage in more abstract and creative thought processes. The paper aims to assess how well current LLMs can tackle these lateral thinking puzzles.

Methods

The authors investigated two main families of pre-trained language models: BERT-based and T5-based models.

  • BERT-based Models: This group includes ALBERT v2, RoBERTa, and DeBERTa v3. The core idea is to fine-tune these models for multiple-choice question answering. For a question with n choices, the model typically performs n forward passes, one for each choice. The output embeddings are then projected to a vector of size n, and a softmax function is used to predict the correct choice. To handle datasets with varying numbers of choices, they normalized questions by either removing extraneous options or adding dummy ones.
  • T5-based Models: This category includes Flan T5 and Unified-QA v2, which are encoder-decoder transformers. These models are fine-tuned to generate the correct choice as a text output, conditioned on the input question.

A key aspect of their methodology was multi-task fine-tuning. They explored fine-tuning models on a combination of the Brain Teaser dataset and various commonsense reasoning datasets. This was hypothesized to provide models with broader knowledge that could aid in the creative reasoning required for lateral thinking puzzles.

Datasets

The primary dataset used for this task is the Brain Teaser dataset, specifically the "Sentence Puzzle" subtask. This dataset consists of original puzzles and adversarial examples (semantic and context reconstructions). The training set contains 169 original puzzles (507 with adversarial examples), and a validation set of 102 examples was used.

To investigate the benefits of multi-task learning, the authors also utilized several auxiliary commonsense reasoning datasets:

  • RiddleSense (RS): For computational creativity.
  • CommonSenseQA (CSQA): For general commonsense knowledge.
  • SWAG: For general commonsense knowledge.
  • HellaSWAG: For general commonsense knowledge.
  • Social IQA (SIQA): For social psychology knowledge.
  • Physical IQA (PIQA): For physical knowledge.

For datasets not inherently in a multiple-choice format (SWAG, HellaSWAG, SIQA), they adapted them into a multiple-choice format for fine-tuning.

Key Insights and Ideas

  • Lateral Thinking is a Distinct Challenge: The Brain Teaser benchmark highlights that lateral thinking puzzles require a different kind of reasoning than standard commonsense tasks, often involving defying typical assumptions.
  • Multi-task Fine-tuning Boosts Performance: Fine-tuning on a combination of Brain Teaser and other commonsense datasets consistently improved model performance compared to training solely on Brain Teaser. This suggests that broader knowledge acquisition is beneficial for creative reasoning.
  • RiddleSense's Impact: Despite being the smallest auxiliary dataset, RiddleSense showed a significant positive impact, likely due to its overlap in dealing with computational creativity and its origin from public websites.
  • DeBERTa-v3's Strength: The DeBERTa-v3-large model emerged as a top performer, achieving an impressive overall accuracy and even surpassing human performance on some metrics. This points to its architectural advantages for this type of task.
  • T5's Advantage with "None of Above": T5-based models, particularly Unified-QA-v2, showed an advantage in handling questions where the correct answer is "None of above," likely because their text-to-text generation allows them to consider all choices simultaneously, unlike the sequential processing of BERT-based models.

Key Results

  • The top-performing model, DeBERTa-v3-large, achieved an overall accuracy of 93.33% on the official test set, surpassing human performance (91.98%) as reported by Jiang et al. (2023).
  • The submission ranked 7th in the official SemEval 2024 Task 9 leaderboard.
  • Multi-task fine-tuning on a combination of Brain Teaser and commonsense datasets (like RS, CSQA, PIQA) generally led to higher accuracies than fine-tuning on Brain Teaser alone. For instance, a Flan T5-base model fine-tuned on a union of datasets achieved 75.53% CV accuracy, while one fine-tuned only on Brain Teaser achieved 75.53% (Table 2).
  • A notable gap was observed between the performance of DeBERTa-v3 on original/semantic puzzles (97.5%) and context-reconstruction puzzles (85.0%), indicating a challenge in generalizing reasoning paths to new situational contexts. Unified-QA-v2 showed a narrower gap (92.5% vs. 87.5%).

Limitations and Future Work

  • Limited Model Size: The study was constrained to models with fewer than one billion parameters due to resource limitations. Future work could explore larger models.
  • Limited Training Data: The Brain Teaser dataset is relatively small, which can lead to noisy results and potential overfitting. The authors used k-fold cross-validation to mitigate this.
  • Dataset Combinations: While multi-task learning was beneficial, the authors limited their final fine-tuning to a combination of Brain Teaser and RiddleSense for computational feasibility. Future work could explore more extensive combinations of auxiliary datasets.
  • Focus on Sentence Puzzles: The study focused solely on the "Sentence Puzzle" subtask. Future research could extend to the "Word Puzzle" subtask.
  • Exploring Other Aspects: The authors suggest future research could explore other aspects of computational creativity and question-answering, potentially with larger models and more diverse training data.

Summary updated

BibTeX
@inproceedings{farokh2024alf,
  author = {Seyed Ali Farokh and Hossein Zeinali},
  title = {{ALF at SemEval-2024 Task 9: Exploring Lateral Thinking Capabilities of LMs through Multi-task Fine-tuning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure