yangqi at SemEval-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges

Qi Yang, Jingjie Zeng, Liang Yang, Hongfei LinNAACL 2024

TL;DR

This paper introduces SHTL, a novel system designed to simulate human lateral thinking for "brain teaser" puzzles that defy common sense. By combining retrieval-augmented generation (RAG) with self-adaptive in-context learning (SAICL) and carefully crafted prompts, SHTL significantly outperforms ChatGPT and closely matches human performance on the SemEval-2024 Task 9 dataset.

What Problem Does This Paper Address?

Human reasoning can be broadly categorized into vertical (logical, convergent) and lateral (creative, divergent) thinking. While natural language processing (NLP) has seen significant progress in tasks requiring vertical thinking, lateral thinking challenges, which often involve defying common sense associations, have been relatively underexplored. The SemEval-2024 Task 9, "BRAINTEASER: A Novel Task Defying Common Sense," aims to bridge this gap by testing models' ability to engage in lateral thinking on a purely English dataset. A key challenge in this domain is the tendency of Large Language Models (LLMs) to "hallucinate" or generate factually incorrect information, which is particularly problematic when dealing with unconventional reasoning.

Methods

The SHTL system operates in two stages:

  1. Common Sense Reasoning: This stage uses an LLM as the core to infer the unreasonable aspects of a brain teaser. To combat hallucinations, it employs two key modules:
  2. Retrieval Augmented Generation (RAG): This module enhances the LLM's understanding by retrieving relevant information from a vast array of unstructured documents and structured knowledge graphs. This retrieved information is then integrated to produce more accurate and richer reasoning.
  3. Self-Adaptive In-Context Learning (SAICL): Inspired by prior work, this module adaptively selects the most relevant in-context examples from the training set for each specific brain teaser. It uses a top-K method for initial selection and then a Minimum Description Length (MDL) principle to organize these examples optimally for the LLM.
  4. Rationalize Defying Common Sense: In this stage, the system combines the common sense reasoning generated in the first stage with the provided multiple-choice options. Carefully designed prompts are used to evaluate which option can logically reconcile the identified common sense violations. If none of the options provide a satisfactory rationale, the system defaults to selecting "None of the above."

Datasets

The primary dataset used for evaluation is the SemEval-2024 Task 9 "BRAINTEASER" dataset. This dataset consists of two sub-tasks: Sentence Puzzles and Word Puzzles, each with original and adversarial versions (Semantic Reconstruction and Context Reconstruction). The paper also mentions using a training set for the SAICL module's in-context example selection.

Key Insights and Ideas

  • Lateral thinking is a distinct challenge: The paper highlights that lateral thinking puzzles require a different kind of reasoning than traditional logical tasks, often involving breaking conventional assumptions.
  • RAG and SAICL are crucial for mitigating LLM hallucinations: Combining retrieval with adaptive in-context learning proves effective in grounding LLM responses and improving their accuracy on complex reasoning tasks.
  • Prompt engineering matters significantly: The authors emphasize that the performance of LLMs can vary greatly based on the specific prompt words used, leading them to develop and select optimal prompt sets.
  • Human-like reasoning is achievable: The system's performance, closely matching human evaluation, suggests that LLMs, when properly guided, can simulate sophisticated human reasoning processes.

Key Results

The SHTL system achieved competitive results on the SemEval-2024 Task 9. On the test set, it demonstrated an average performance that was only 0.8 points lower than human evaluation and significantly outperformed ChatGPT in a zero-shot setting. Specifically, SHTL achieved 90.0% instance-based accuracy on the Original Sentence Puzzle and 90.6% on the Original Word Puzzle, with comparable or higher scores on adversarial subsets.

Limitations and Future Work

  • Limitations: The paper acknowledges that while SHTL performs well, there's still a slight gap compared to human performance. The reliance on LLMs means that the system is still susceptible to the inherent limitations of these models, though RAG and SAICL help mitigate this.
  • Future Work: The authors suggest that LLMs' latent linguistic capabilities need further exploration and appropriate utilization. This implies a direction towards developing more sophisticated prompting strategies and potentially more advanced integration of external knowledge sources.

Summary updated

BibTeX
@inproceedings{yang2024yangqi,
  author = {Qi Yang and Jingjie Zeng and Liang Yang and Hongfei Lin},
  title = {{yangqi at SemEval-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure