Abstract
Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on a heavily curated, language-independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state-of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages.
Introduction
Most progress in Question Answering (QA) is measured in English. While multilingual datasets exist, they often suffer from three major flaws: limited language diversity, "translationese" artifacts from translated passages, and heavy dependency on specific text spans (extractive QA). This makes them unsuitable for evaluating open-domain systems that utilize Knowledge Graphs or generative models. MKQA addresses these gaps by providing a realistic, geographically invariant, and linguistically diverse evaluation set that decouples answers from specific source passages.
Methodology
The researchers developed MKQA using a rigorous six-step pipeline designed to ensure high quality and cross-lingual alignment.
The process begins with sampling 10,000 queries from the Natural Questions (NQ) dataset. Unlike the original NQ, which relies on specific Wikipedia passages, MKQA involves five independent annotators researching each query to find the "ideal" answer. These answers are then grounded in Wikidata QIDs, allowing for a language-independent representation. Finally, professional translators localize the queries into 25 additional languages, ensuring natural phrasing while preserving intent.
Dataset Characteristics
MKQA represents a significant expansion in scale and diversity compared to previous multilingual QA benchmarks. It covers 26 languages across 14 language families, totaling 260,000 question-answer pairs.
A key innovation in MKQA is its "retrieval-independence." By grounding answers in Wikidata entities, the benchmark can evaluate generative models and Knowledge Graph systems alongside traditional retrieval-based models. Furthermore, the curation process revealed that 70.4% of questions marked "unanswerable" in the original NQ dataset actually have valid, findable answers, making MKQA a more rigorous test of a model's true knowledge.
Results and Discussion
Benchmarking state-of-the-art models like XLM-R, mBERT, and mT5 reveals a significant performance gap. The best-performing model (Translate-Train XLM-R) achieved a macro-averaged F1 score of only 46.0 across all languages. English performance remains the highest (52.3 F1), while low-resource languages like Khmer and Arabic lag significantly behind. Generative models showed promise in "closed-book" settings but still struggle to match models with access to English Wikipedia passages.
Conclusion
MKQA establishes a new gold standard for multilingual open-domain QA. While it currently relies on English-origin queries, it provides the most diverse evaluation set to date. Future work will explore "Generalized Cross-lingual Transfer" (G-XLT), where models must retrieve answers from corpora in languages different from the query.
Summary updated