TL;DR
CulturalCrosswords is a new multilingual benchmark that evaluates how Large Language Models (LLMs) handle cultural riddles under varying constraints like answer length and grid crossings. The study reveals that while constraints significantly boost "exact-match" scores by helping models select the correct form of an answer, there remains a massive gap between a model's ability to solve a clue and its ability to correctly place that answer within a spatial grid.
What Problem Does This Paper Address?
Evaluating LLMs on "cultural knowledge" is notoriously tricky. If a riddle asks for a popular soft drink, "Coke" and "Coca-Cola" are both semantically correct, but a standard evaluation script might mark one as wrong if it doesn't perfectly match the reference. This is the exact-match penalty.
Beyond simple knowledge retrieval, real-world problem-solving often involves constraints. In a crossword, an answer isn't just a word; it must be exactly $N$ letters long and share specific letters with intersecting words. The authors tackle two main questions:
- How do these constraints help models disambiguate between multiple valid answers?
- Can LLMs maintain "crossing consistency"—ensuring that the "S" in an Across answer is the same "S" required by the Down answer?
Methods
The researchers evaluated 11 LLM configurations (including GPT-6 Astra, Gemini 4 Argon, and Claude Opus 5.5) across five distinct task conditions to isolate the effects of different information types:
- Condition A (Plain): A batch of clues with no extra info.
- Condition B (Length): Clues plus the required answer length.
- Condition C (Hints): Lengths plus partial-answer hints (e.g., every 4th letter provided).
- Condition D (Grid): The model must return a completed 50x50 text grid.
- Condition E (Crossword/List): The model gets the full grid structure and coordinates but returns a simple list of answers.
This setup allows the authors to distinguish between answer-form selection (choosing "Coke" because it's 4 letters) and semantic disambiguation (choosing "Coke" because the hints rule out "Pepsi").
Datasets
The authors introduced the CulturalCrosswords benchmark, constructed from the existing CulturalRiddles dataset.
- Scope: 53 unique crosswords.
- Diversity: Covers 42 languages and 42 countries.
- Complexity: 5,057 total entries across various writing systems, including alphabets, abjads (Arabic), abugidas (Indic), and Japanese katakana.
Key Insights and Ideas
- The "Form" vs. "Meaning" Distinction: A huge portion of the performance gain seen when adding constraints comes from "answer-form selection." Models often know the right entity but need the length constraint to provide the specific string the evaluator expects.
- Crossings as Logic Checks: Crossword structure improves "compatibility." Even if a model doesn't get more answers right compared to the reference, the answers it does provide are more likely to actually fit together in the grid.
- The Spatial Reasoning Gap: There is a stark difference between solving a list of clues and filling a grid. Some models that are highly accurate at solving clues fail miserably when asked to output the final 50x50 grid (Condition D), suggesting that spatial reasoning and grid-adherence are distinct skills from linguistic reasoning.
Key Results
- Constraint Impact: Adding answer lengths (Condition B) improved agreement with the reference by 22.8 percentage points. However, when the scoring was relaxed to accept valid alternatives, that gain dropped to 8.4 points, proving that much of the "improvement" was just the model guessing the right word format.
- Consistency: Providing the crossword structure (Condition E) increased the share of compatible crossings by 9.7 points over just providing lengths.
- Model Performance: GPT-6 Astra and Gemini 4 Argon emerged as top performers. Astra excelled at plain batches and grid completion, while Argon led in the length and hint conditions.
- The Grid Failure: For example, Gemini 3.7 Flash scored 75.6% on the crossword answer list but plummeted to 5.4% when required to actually fill the grid.
Limitations and Future Work
- Incomplete Alternatives: The list of "accepted alternative answers" used for scoring is not exhaustive, meaning the true semantic accuracy of the models might be even higher than reported.
- Confounded Variables: Because the puzzles are tied to specific languages and cultures, it's hard to tell if a model is struggling with the language or the cultural content of the riddle.
- Instruction Sensitivity: The authors noted that different models responded differently to the "Grid" instructions, suggesting that prompt engineering plays a massive role in spatial tasks.
- Future Direction: The authors suggest applying this "constraint-based" evaluation to other short-answer benchmarks to see if LLMs are generally being penalized for valid but non-matching answers.
Summary updated