TL;DR
MKQA (Multilingual Knowledge Questions and Answers) is a massive new benchmark featuring 10,000 question-answer pairs aligned across 26 diverse languages. By grounding answers in language-independent Wikidata entities rather than specific text passages, it provides a truly "retrieval-independent" way to evaluate how well AI models understand and answer questions globally.
What Problem Does This Paper Address?
Most progress in Question Answering (QA) is measured in English. While multilingual datasets exist, they often suffer from three major flaws:
- Lack of Diversity: They usually cover only a handful of high-resource languages.
- "Translationese" Artifacts: Many datasets are created by translating English passages, which leaves behind linguistic cues that make the task artificially easy for models.
- Passage Dependency: Most benchmarks are "extractive," meaning the answer must be a specific span of text within a provided document. This doesn't work for "open-domain" systems that might use a Knowledge Graph (like Wikidata) or a generative model (like GPT or T5) to find answers.
The authors introduce MKQA to bridge this gap, providing a realistic, geographically invariant, and linguistically diverse evaluation set that doesn't care how a model finds the answer, only that the answer is correct.
Methods
The researchers built MKQA using a rigorous six-step pipeline:
- Query Selection: They sampled 10,000 real-world queries from the "Natural Questions" (NQ) dataset—questions actually typed by users into search engines.
- Answer Curation: Instead of just using the original NQ answers, five annotators independently researched each question to find the "ideal" answer.
- Wikidata Grounding: Answers were linked to Wikidata QIDs (unique identifiers). This is the "secret sauce"—it allows the answer to be represented as a concept (e.g., "Q794" for Iran) rather than just a string of text, making it easy to translate the answer into hundreds of other languages automatically.
- Human Translation: Professional bilingual translators localized the 10,000 English queries into 25 other languages, ensuring they felt natural and preserved the original meaning.
- Quality Control: They enforced strict inter-annotator agreement and performed manual audits to ensure the translations and answers were accurate.
Datasets
- MKQA (New): 260,000 total QA pairs (10k per language) across 26 languages and 14 language families.
- Natural Questions (NQ): Used as the source for the initial 10k English queries.
- Wikidata: Used for grounding answers and providing multilingual aliases for entities.
Key Insights and Ideas
- Retrieval-Independence is Key: By decoupling the answer from a specific passage, MKQA can evaluate generative models (which "know" facts in their weights) and Knowledge Graph systems alongside traditional search-and-extract models.
- Fixing the "Unanswerable" Label: In the original NQ dataset, many questions were marked "unanswerable" simply because the answer wasn't in the specific Wikipedia passage provided. MKQA researchers found that 70.4% of these "unanswerable" NQ questions actually have valid, findable answers. This makes MKQA a much more challenging and realistic test of a model's true knowledge.
- Geographical Invariance: The authors specifically selected queries where the answer doesn't change based on where you are (e.g., "Who wrote War and Peace?" vs. "Who is my local representative?"). This ensures a fair comparison across all 26 languages.
Key Results
The authors benchmarked several state-of-the-art models, including XLM-R, mBERT, and generative mT5 models.
- The Performance Gap: Even the best model (a "Translate-Train" XLM-R) only achieved a 46.0 F1 score macro-averaged across all languages.
- English is Still King: The best English performance was 52.3 F1, while low-resource languages like Khmer and Arabic lagged significantly behind.
- Generative vs. Extractive: Generative models (like mT5) performed surprisingly well in "closed-book" settings (answering without any external documents), but still struggled to match models that had access to English Wikipedia passages.
- Baseline Comparison: A naive baseline that simply predicts "No Answer" for every question gets a 32.4 F1 score, highlighting how difficult the "answerable" portion of the dataset really is.
Limitations and Future Work
- English Centricity: While the queries are translated into 26 languages, they all originated as English search queries. This might miss types of questions that are unique to other cultures or languages.
- Static Answers: Some answers (like "Who is the current president?") change over time. While the authors tried to ground these, the dataset will eventually require updates.
- Future Directions: The authors suggest using MKQA for "Generalized Cross-lingual Transfer" (G-XLT), where the question is in one language and the answer must be retrieved from a corpus in a different language.
Summary updated