TL;DR
This paper explores three unsupervised models for solving riddles, a task requiring both logical and creative "lateral" thinking. While none of the models achieved top performance, the best approach, a bag-of-words model leveraging word sense disambiguation, showed promise by outperforming a baseline and securing a respectable 10th place in a competitive task. This work highlights the potential of WSD in tackling nuanced language understanding challenges.
What Problem Does This Paper Address?
Riddles are notoriously tricky for AI because they often rely on "commonsense" knowledge and lateral thinking – the ability to think outside the box and consider unconventional interpretations. Standard language models, which excel at logical, sequential reasoning (vertical thinking), can be easily misled by the misdirection and wordplay inherent in riddles. The BRAINTEASER task (SemEval-2024 Task 9) specifically challenges models to solve riddles by choosing the correct answer from a set of four options, aiming to bridge the gap between vertical and lateral thinking in AI. The dataset for this task was compiled from public websites, presenting a real-world challenge for AI systems.
Methods
The authors developed three unsupervised models, all built upon the bert-large-cased language model.
- Bag-of-Words with WSD (BOW-WSD): This was their best-performing model. It first identifies nouns in the riddle and its potential answers, then uses WordNet and SensEmBERT to determine the most likely sense for each noun. These senses are then used to create "bags of words" for the question and each answer. The similarity between these bags (measured by the overlap of common words) is used to predict the correct answer.
- Language Modelling Approach: This method treats riddle-solving as a text completion task. It concatenates each potential answer to the riddle's question and uses the
bert-large-casedmodel to calculate the probability of the resulting text. The answer that yields the highest probability is chosen. - Sense Comparison Approach: This model also leverages WordNet senses for nouns. It calculates the cosine similarity between the embeddings of the [CLS] token for each identified sense in the question and each sense in the potential answers. The answer with the highest aggregated similarity score is predicted.
A key aspect of their approach is the use of unsupervised learning, meaning they did not rely on labeled training data for the BRAINTEASER task itself, making their models potentially more adaptable.
Datasets
The primary dataset used was the BRAINTEASER dataset provided for SemEval-2024 Task 9. This dataset consists of riddles presented as questions with four possible answers. Each sample also includes semantic and context reconstructions to further test reasoning abilities. The authors also utilized WordNet (a lexical database) and English stop words from the NLTK Python library.
Key Insights and Ideas
- WSD is a promising avenue for riddle-solving: The BOW-WSD model's performance suggests that understanding the specific senses of words, especially nouns, can help cut through the misdirection in riddles.
- Lateral thinking is hard to capture: Despite efforts to incorporate WSD and language modeling, the models still struggled to consistently replicate the "aha!" moments of human riddle-solvers.
- Language modeling excels at certain riddle structures: The language modeling approach performed surprisingly well on word puzzles, likely because their structure as questions lends itself better to text completion.
- Unsupervised learning has limitations: While valuable for adaptability, the lack of task-specific training data meant their models couldn't compete with supervised baselines or human performance.
Key Results
The BOW-WSD model, their best performer, achieved 10th place out of 11 teams on both the sentence puzzle and word puzzle subtasks. It managed to surpass the RoBERTa-L baseline in several categories for both sentence and word puzzles, demonstrating its effectiveness against a strong baseline. The language modeling approach also outperformed the RoBERTa-L baseline in some word puzzle categories. However, all their unsupervised models fell significantly short of the ChatGPT and Human baselines.
Limitations and Future Work
The authors acknowledge that their unsupervised models did not perform as well as supervised approaches or human baselines. They suggest that the "red herrings" in riddles can still significantly impact their BOW-WSD model.
Future work could involve:
- Ensemble methods: Combining multiple approaches to leverage their respective strengths.
- Improved WSD integration: Exploring other ways to incorporate WSD into their riddle-answering models.
- Penalizing sense divergence: Adding a metric to penalize answers whose noun senses are wildly different from those in the question, specifically to counter the effect of misleading clues.
Summary updated