BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles

Yunxiang Zhang, Xiaojun Wan

Abstract

BiRdQA is a novel bilingual dataset comprising over 15,000 English and Chinese riddles, specifically curated to test the limits of figurative language comprehension and commonsense reasoning in large language models (LLMs). Unlike standard QA benchmarks that focus on literal fact retrieval, BiRdQA requires models to navigate metaphors, puns, and personification. Our evaluation shows that even state-of-the-art models lag behind human performance by up to 30%, underscoring a critical deficiency in current AI's ability to interpret veiled meanings and creative wordplay.

The Challenge of Figurative Language

Most existing Question Answering (QA) benchmarks, such as SQuAD or HotpotQA, rely on literal, fact-based information sourced from Wikipedia or news articles. While these test a model's ability to retrieve information or perform multi-hop logic, they rarely challenge a model's grasp of figurative language—including metaphors, puns, and personification—or its ability to handle misleading descriptions.

Riddles serve as an ideal testbed for these capabilities. For example, an English riddle like "As far as I run, I never meet my brother on my right. What am I?" (Answer: a wheel) requires a model to realize that "brother" is a personification of a parallel wheel and "run" refers to mechanical rotation. Current models often fall into the trap of "spurious correlations," selecting answers based on common word associations rather than reasoning through the puzzle's logic.

Examples of English and Chinese riddles in BiRdQA, illustrating the use of figurative language and the inclusion of knowledge snippets for candidate answers.

The BiRdQA Dataset

The BiRdQA dataset consists of two independent, non-parallel subsets: BiRdQA-en (6,614 English riddles) and BiRdQA-zh (8,751 Chinese riddles). Because riddles are deeply rooted in specific linguistic and cultural contexts—particularly puns and character-based puzzles—the authors crawled and cleaned these subsets independently to preserve their inherent difficulty.

Statistical breakdown of the BiRdQA-en and BiRdQA-zh subsets, including example counts and token distributions across training, validation, and test sets.

Methodology

The authors framed riddle-solving as a five-way multiple-choice task. To ensure robustness and prevent models from "cheating" via simple pattern matching, they employed a sophisticated distractor generation strategy:

  1. Global Distractors: Two choices are generated by finding words semantically similar to the correct answer using word embeddings.
  2. Local Distractors: Two choices are drawn from the pool of correct answers for other riddles in the dataset, ensuring the model cannot learn a specific "riddle answer" distribution.
  3. Knowledge Augmentation: Recognizing that riddles require external knowledge, every candidate answer is linked to its introductory paragraph from Wikipedia (English) or Baidu Baike (Chinese).
  4. Invalid Replacement: Strict rules were implemented to prevent lexical overlap between distractors and the correct answer (e.g., avoiding "horse" and "wild horse" in the same set).

Key Results and Insights

  • The Human-Machine Gap: Humans achieve approximately 81-87% accuracy on BiRdQA, while the best-performing models—UnifiedQA (T5-Large) for English (62.60%) and ERNIE for Chinese (59.29%)—lag significantly behind. This ~25% gap represents the remaining "reasoning headroom" for NLP research.
  • Prevalence of Figurative Language: Manual analysis revealed that 31% of English and 44% of Chinese riddles in the sample used at least one figure of speech. Personification was most common in English, while puns and metaphors led in the Chinese subset.
  • Cross-Lingual Transfer: Training on a mix of English and Chinese data improved performance on English riddles, suggesting that models can transfer some underlying "riddle-solving logic" across languages despite the lack of parallel text.
  • Knowledge Utility: While Wikipedia descriptions boosted performance for English riddles, they had minimal impact on Chinese results. This suggests that Chinese riddles may require deeper character-level semantic analysis rather than literal encyclopedic facts.

Limitations and Future Work

The English subset is currently smaller and less diverse than the Chinese subset. Furthermore, the Chinese subset excludes "character riddles" (puzzles based on Hanzi structure), which represent a significant cultural component but require specialized stroke-level modeling. Future work will focus on expanding BiRdQA to more languages and developing architectures that explicitly model figurative language and cultural nuances.

Summary updated

Abstract

A riddle is a question or statement with double or veiled meanings, followed by an unexpected answer. Solving riddle is a challenging task for both machine and human, testing the capability of understanding figurative, creative natural language and reasoning with commonsense knowledge. We introduce BiRdQA, a bilingual multiple-choice question answering dataset with 6614 English riddles and 8751 Chinese riddles. For each riddle-answer pair, we provide four distractors with additional information from Wikipedia. The distractors are automatically generated at scale with minimal bias. Existing monolingual and multilingual QA models fail to perform well on our dataset, indicating that there is a long way to go before machine can beat human on solving tricky riddles. The dataset is publicly available at https://forms.gle/NvT7DfWhAPhvoFvH7.
BibTeX
@inproceedings{zhang2022birdqa,
  author = {Yunxiang Zhang and Xiaojun Wan},
  title = {{BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21430},
}
Figure