The Shared Word: Evaluating Language Models on Dutch Word Puzzles

Rik van Noord, Maurice VoorsarXiv 2026 · arXiv

TL;DR

The authors introduce The Shared Word (TSW-NL), a challenging Dutch benchmark requiring LLMs to solve word puzzles by identifying a single word described by three distinct, ambiguous clues. The results reveal a massive performance gap: while the best open-weight models struggle to reach 40% accuracy, state-of-the-art reasoning models like GPT-5.6 Sol achieve nearly 89%, significantly outperforming human annotators.

What Problem Does This Paper Address?

Evaluating Large Language Models (LLMs) in languages other than English is often a flawed process. Most existing benchmarks are either direct translations of English datasets—which lose cultural and linguistic nuances—or rely on multiple-choice formats that allow models to "guess" without truly understanding. Furthermore, generative tasks often suffer from "fuzzy" evaluation metrics that make objective comparison difficult.

The authors argue that a "good" non-English benchmark should be:

  1. Language-specific: The core logic shouldn't survive translation.
  2. Open-vocabulary: The model must generate the answer, not pick from a list.
  3. Human-curated: To ensure quality and avoid "garbage in, garbage out."
  4. Deterministically scored: Using exact-match criteria for objective results.

TSW-NL is designed specifically to meet these four criteria for the Dutch language.

Methods

The task is inspired by the Dutch TV quiz Met Het Mes Op Tafel. A model is given three clues that describe different senses or uses of the same word. For example, as shown in Figure 1, the clues "a football coach," "an alcoholic drink," and "a profession" all point to the Dutch word advocaat.

Figure 1: Example puzzle. The three Dutch clues de- scribe a football coach (Dick Advocaat), an alcoholic drink (Dutch-specific, similar to eggnog), and a profes- sion (lawyer), all answered by advocaat.

To build the dataset, the researchers:

  • Scraped Wikipedia: They used Dutch Wikipedia disambiguation pages as a starting point.
  • Automated Filtering: They filtered for common words (using Open Dutch WordNet), removed short abbreviations, and ensured the source pages were sufficiently popular (at least 3,000 views/month).
  • Manual Curation: Two curators meticulously reviewed the candidates to ensure that no single clue gave away the answer and that the three clues collectively pointed to exactly one intended word.
  • Evaluation: Models were tested in a zero-shot setting. To account for clue-order sensitivity, every model was evaluated on all six possible permutations of the three clues.

Datasets

The paper introduces TSW-NL, which consists of 222 instances (22 for development and 200 for the final test set).

The authors compared TSW-NL against existing Dutch benchmarks like DUMB, EuroEval, and BEIR-NL. They found that none of these existing tasks satisfied all four of their core principles (Language-specific, Open-vocabulary, Human-verified, Deterministic evaluation).

Key Insights and Ideas

  • Evidence Integration is Crucial: The task is designed so that clues are individually ambiguous. As shown in Figure 2, model accuracy jumps significantly when moving from one clue to two, and again from two to three. This confirms that the benchmark successfully tests a model's ability to combine disparate pieces of evidence.
  • Reasoning Models Excel: Models with explicit "reasoning" or "thinking" steps (like the GPT-5.6 "Sol" variants) performed dramatically better. This suggests the task isn't just about retrieval; it requires generating candidates and internally verifying them against all clues.
  • The "Dutch-Specific" Advantage is Small: While models specifically fine-tuned on Dutch (like GEITje) performed better than their direct multilingual parents (like Mistral), they still lagged far behind larger multilingual models and proprietary systems.
  • Humans are Fallible: Human annotators averaged only 54% accuracy. This highlights that the benchmark is genuinely difficult even for native speakers, often due to the breadth of general knowledge required.
Figure 2: Model accuracy by number of clues. For clar- ity, only a few models are highlighted in color. The other models are shown in grey. One- and two-clue scores average over the three corresponding clue subsets.

Key Results

The evaluation covered a wide range of models, from small Dutch-specific ones to massive proprietary systems.

  • Human Baseline: 54.0% accuracy.
  • Open-Weight Models: Most struggled. The top performer was Gemma 3 27B IT at 39.3%. Many smaller models scored below 20%.
  • Proprietary Models: Showed significant "headroom." GPT-4o scored 56.3%, while GPT-5.6 Sol (max reasoning) reached a staggering 88.8%.

The full results (summarized in the paper's tables) show a clear hierarchy where scale and reasoning capabilities are the primary drivers of success on this task.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: At 200 test items, the benchmark is relatively small, which can limit statistical power.
  • Wikipedia Exposure: Since the data originates from Wikipedia, models may have seen these associations during pre-training. However, the low scores of most models suggest that simple "memorization" isn't enough to solve the puzzles reliably.
  • Subjectivity: Manual curation involves subjective judgments about what constitutes an "obscure" reference.

Future work could involve applying this "recipe" to other languages with robust Wikipedia disambiguation pages to create a suite of truly language-specific generative benchmarks.

Summary updated

Abstract

Many LLM benchmarks for languages other than English are translated, multiple-choice, or difficult to score automatically. We introduce The Shared Word (TSW-NL), a Dutch benchmark built around four core principles: it is language-specific, open-vocabulary, human-curated, and deterministically scored. Each puzzle gives three clues that describe different senses or uses of the same Dutch word. The clues are individually ambiguous, requiring the solver to combine evidence across multiple clues. We construct the benchmark from Dutch Wikipedia disambiguation pages, followed by automatic filtering and extensive manual curation. The final dataset contains 222 instances, with three independent human answers per test item. The task is clearly challenging: scores in our human study range from 49.5% to 61.0% and the highest-scoring open-weight model we tested reaches only 39.3%. Many others even score below 20%. At the same time, closed-source models show great progress: GPT-4.1 (April 2025) scores 59.2%, whereas GPT-5.6 Sol with max reasoning scores 88.8%. We believe this task provides a useful addition to Dutch model evaluation, plus a clear recipe that can be applied to construct a similar benchmark for other languages.
BibTeX
@misc{noord2026shared,
  author = {Rik van Noord and Maurice Voors},
  title = {{The Shared Word: Evaluating Language Models on Dutch Word Puzzles}},
  year = {2026},
  eprint = {2611.22935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22935},
}
Figure