FtG-CoT at SemEval-2024 Task 9: Solving Sentence Puzzles Using Fine-Tuned Language Models and Zero-Shot CoT Prompting

Micah Zhang, Shafiuddin Rehan Ahmed, James H. MartinNAACL 2024

TL;DR

This paper introduces FtG-CoT, a novel prompting technique that combines fine-tuned BERT encoders with zero-shot Chain-of-Thought (CoT) generation and a fine-tuned LLM to solve challenging sentence puzzles. FtG-CoT significantly outperforms a baseline zero-shot approach, achieving a perfect score on the practice set and a strong 0.9 score on the evaluation set for the BRAINTEASER task.

What Problem Does This Paper Address?

The BRAINTEASER task (SemEval-2024 Task 9) aims to evaluate the "lateral thinking" or "thinking outside the box" capabilities of large language models (LLMs). This is a departure from traditional NLP tasks that rely on common sense. These puzzles are designed to defy default commonsense associations, requiring creative thinking and the ability to challenge preconceptions. The task includes sentence puzzles (narrative-based) and word puzzles (letter-composition based), both presented in a multiple-choice format. The authors focus on the sentence puzzle sub-task, which is particularly challenging as it requires understanding nuanced interpretations and avoiding superficial reasoning.

Methods

The core contribution is the Fine-tuned Generated Chain-of-Thought (FtG-CoT) prompting method, which involves five key steps:

  1. Fine-tune BERT Encoder: A pre-trained BERT-Small model is fine-tuned as a multi-class classifier on the training data. This encoder is trained to capture contextual and semantic information relevant to solving sentence puzzles, producing specialized encodings for questions and answer choices.
  2. Zero-Shot CoT Generation: For each training example, a zero-shot CoT prompt is used to ask an LLM (GPT-3.5) to generate an explanation for why the correct answer is correct. This leverages CoT prompting without manual creation of reasoning chains. The prompt includes the question, choices, correct answer, and a "Let's think step by step" instruction.
  3. Fine-tune LLM: A large language model (GPT-3.5 Turbo) is then fine-tuned on these generated chain-of-thought explanations. This step aims to improve the LLM's accuracy and consistency in generating reasoning.
  4. Rank Demonstrations: The fine-tuned BERT encoder is used to generate vector embeddings for both the test question and all training examples. Cosine similarity is calculated between the test question embedding and each training example embedding.
  5. Construct Few-Shot Prompt: The top N training examples with the highest cosine similarity to the test question are selected. These demonstrations, along with the test question and choices, are combined into a few-shot prompt for the fine-tuned LLM. The demonstrations are ordered by increasing similarity.

A fuzzy logic answer extractor is used to parse the LLM's response and match it to the most similar answer choice.

Datasets

  • BRAINTEASER Training Set: Provided by the task organizers, consisting of 507 sentence puzzles. Each example includes a question, answer choices, the correct answer, and distractors. Adversarial versions (semantic and context reconstruction) were also generated to prevent memorization.
  • BRAINTEASER Practice and Evaluation Sets: Each contains 120 questions for evaluating models.

Key Insights and Ideas

  • Leveraging BERT for Relevance: Fine-tuning BERT as a classifier proves effective in creating an encoder that can identify the most semantically relevant training examples for few-shot prompting. This is crucial for selecting demonstrations that truly aid the LLM.
  • Zero-Shot CoT for Training Data: Generating reasoning chains via zero-shot CoT is a practical way to create training data for fine-tuning LLMs without manual annotation, which would be prohibitively expensive for complex reasoning tasks.
  • Ordered Demonstrations Matter: Presenting few-shot demonstrations in order of increasing similarity to the test question appears to be a more effective strategy than other orderings.
  • Fine-tuning is Key: Fine-tuning the LLM on the generated CoT explanations significantly boosts performance compared to using a pre-trained LLM with just few-shot examples.

Key Results

  • Performance: FtG-CoT achieved a perfect score on the practice set and a 0.9 accuracy score on the evaluation set for the sentence puzzle sub-task.
  • Ranking: The method ranked 18th overall in the SemEval-2024 BRAINTEASER competition.
  • Comparison: FtG-CoT significantly outperformed the zero-shot ChatGPT baseline provided by the task organizers across all six evaluation metrics.
  • Demonstration Count: Performance generally improved with more demonstrations, peaking around 20. Beyond this, performance plateaued or slightly decreased.
  • Fine-tuning Impact: Fine-tuning the LLM led to a substantial performance jump from around 0.925 to 0.975, and a perfect 1.0 score was achieved with 20 demonstrations and fine-tuning.

Limitations and Future Work

  • Sensitivity to Rephrasing: The model showed a notable drop in performance on semantically and contextually reconstructed questions compared to original questions. This suggests a potential sensitivity to word choice and phrasing, though the drop wasn't so large as to indicate pure memorization.
  • Future Work:
    • Data Augmentation: Using LLMs to rephrase existing training questions could expand the training set and potentially reduce sensitivity to phrasing.
  • Human Feedback: Incorporating human feedback via reinforcement learning could further improve the quality of generated reasoning chains and demonstrations. This could involve human review and correction of generated chains.

Summary updated

BibTeX
@inproceedings{zhang2024ftg,
  author = {Micah Zhang and Shafiuddin Rehan Ahmed and James H. Martin},
  title = {{FtG-CoT at SemEval-2024 Task 9: Solving Sentence Puzzles Using Fine-Tuned Language Models and Zero-Shot CoT Prompting}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure