MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thought Prompts

Nishat Raihan, Dhiman Goswami, Al Nahian Bin Emran, Sadiya Sayara Chowdhury Puspo, Amrita Ganguly, Marcos ZampieriNAACL 2024

TL;DR

This paper tackles the challenge of solving complex natural language puzzles using Large Language Models (LLMs). The MasonTigers team found that "chain-of-thought" prompting, which guides LLMs to break down reasoning into steps, significantly improves performance. Their ensemble approach, combining multiple chain-of-thought prompts, achieved top rankings in the SemEval-2024 Task 9 puzzle competition.

What Problem Does This Paper Address?

Natural language understanding (NLU) tasks have seen remarkable progress with LLMs, but complex reasoning, especially in puzzle-solving, remains a significant hurdle. The SemEval-2024 Task 9, featuring the "BrainTeaser" dataset, introduces a benchmark of intricate word and sentence puzzles that require multi-step inference and deduction, often going beyond simple pattern matching or domain knowledge. These puzzles demand the ability to connect disparate pieces of information, understand implicit relationships, and apply logic creatively, pushing the boundaries of current LLM capabilities.

Methods

The MasonTigers team explored several prompting strategies to guide LLMs in solving these puzzles. They started with zero-shot prompting, where the model receives the puzzle and potential answers without any prior examples. This was followed by few-shot prompting, where a few solved examples are provided to give the model context and demonstrate the desired output format.

The core of their approach, however, was chain-of-thought (CoT) prompting. This technique involves explicitly instructing the LLM to "think step-by-step," breaking down the reasoning process into a series of logical deductions. They experimented with varying the number of steps (2-shot, 4-shot, 8-shot) within the CoT framework. To further enhance robustness and confidence in their predictions, they implemented an ensemble method. This involved aggregating the outputs from multiple CoT-prompted models, using majority voting to arrive at the final answer. This ensemble approach aimed to mitigate individual model biases and improve overall accuracy.

Datasets

The primary dataset used in this work is the BrainTeaser dataset, introduced with SemEval-2024 Task 9. This dataset comprises 1,100 multiple-choice questions divided into two sub-tasks:

  • Sentence-based puzzles (627 questions): These rely on narrative context and common phrases.
  • Word-based puzzles (492 questions): These focus on the literal form and letters of words.

The dataset is designed to test lateral thinking and creative reasoning, with each question having one correct answer and three distractors. For evaluation, the task organizers provided development and test sets.

Key Insights and Ideas

  • Chain-of-Thought is Crucial for Complex Reasoning: The paper strongly reinforces that CoT prompting is not just a minor improvement but a fundamental enabler for LLMs to tackle complex reasoning tasks like puzzle-solving. Simply providing examples (few-shot) is less effective than guiding the model's internal reasoning process.
  • Ensembling Boosts Robustness: Combining the outputs of multiple CoT-prompted models through majority voting significantly improved performance and reliability. This suggests that different models or even different CoT prompts might latch onto different aspects of the problem, and an ensemble can synthesize these diverse perspectives.
  • Proprietary LLMs Outperform Open-Source (in this context): The results indicate that state-of-the-art proprietary models like GPT-4 Turbo and Claude 2.1 generally outperformed the open-source Mixtral model on this specific task, especially when employing advanced prompting techniques.
  • Prompt Specificity Matters: The authors observed that performance improved with more specific and detailed chain-of-thought prompts, highlighting the importance of prompt engineering.

Key Results

The MasonTigers team achieved competitive results on the SemEval-2024 Task 9 leaderboard. Their best performance, utilizing an ensemble of 8-shot chain-of-thought prompts, secured them 2nd place in the word puzzle sub-task and 13th place in the sentence puzzle sub-task. Specifically, GPT-4 Turbo with the ensemble method reached scores of 0.93 (Sentence Dev), 0.93 (Sentence Test), 0.96 (Word Dev), and 0.95 (Word Test). Claude 2.1 with the ensemble achieved 0.91 (Sentence Dev), 0.86 (Sentence Test), 0.91 (Word Dev), and 0.95 (Word Test). While Mixtral's performance was lower, it also benefited from CoT and ensemble methods.

Limitations and Future Work

The authors acknowledge several limitations:

  • Human Effort in Prompt Engineering: Crafting effective prompts is labor-intensive and requires significant human expertise. Automating this process is a key area for future research to improve scalability.
  • Performance Gap with Humans: Despite improvements, LLM performance still lags behind human levels on these complex reasoning tasks, indicating room for advancement.
  • Architectural Constraints: Limitations related to long-term memory and reasoning within LLM architectures may need to be addressed.
  • Generalizability of Reasoning: The current approach focuses on solving specific puzzles rather than teaching broader inferential skills. Developing methods for more generalizable reasoning through prompting remains an open challenge.

Summary updated

BibTeX
@inproceedings{raihan2024masontigers,
  author = {Nishat Raihan and Dhiman Goswami and Al Nahian Bin Emran and Sadiya Sayara Chowdhury Puspo and Amrita Ganguly and Marcos Zampieri},
  title = {{MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thought Prompts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure