TL;DR
Current multilingual QA benchmarks assume that translating a question shouldn't change its answer. This paper introduces XLQA, a new benchmark that explicitly accounts for cultural and regional variations across eight languages, revealing that state-of-the-art LLMs frequently fail when answers depend on local context.
What Problem Does This Paper Address?
Most multilingual open-domain question answering (ODQA) benchmarks rely on a flawed assumption: locale invariance. They assume that if you translate a question like "How long did it take the Twin Towers to be built?" into Korean or Chinese, the answer should still refer to the World Trade Center in New York.
In reality, a speaker in Seoul asking about the "Twin Towers" is likely referring to the LG Twin Towers, while a speaker in China might mean the Tianjin IFC.
By ignoring these regional and cultural nuances, current evaluation metrics suffer from a severe bias—they penalize language models for providing answers that are locally accurate but differ from the original English ground truth.
Methods
To build a benchmark capable of evaluating this "locale awareness" without requiring prohibitive amounts of manual labor, the authors developed a scalable, three-stage pipeline:
- Multilingual Question Generation: English seed questions were translated into seven target languages using GPT-4. To eliminate translation artifacts and "semantic drift" (subtle shifts in meaning), they implemented a back-translation filtering step. Questions were translated back into English, and an LLM-as-judge evaluated whether the original and back-translated questions were semantically identical.
- Locale-Aware Answer Generation: The translated questions were fed into an LLM connected to a retrieval-augmented generation (RAG) web search component. The model was prompted to find answers grounded in language-specific sources (like localized Wikipedia). If the generated answer diverged semantically from the English baseline, it was flagged as a potential "knowledge conflict."
- Human Verification: Human annotators (via Amazon Mechanical Turk) reviewed the flagged questions, answers, and supporting evidence using a majority-voting scheme to ensure absolute correctness and verify whether the question was truly locale-sensitive.
Datasets
- Seed Data: 3,000 high-quality English questions filtered from existing benchmarks (MKQA, MLQA, and HotpotQA).
- XLQA (New Benchmark): A dataset of 24,000 question-answer-evidence triples spanning 8 languages: English, Korean, Arabic, Hebrew, Japanese, Russian, Vietnamese, and Simplified Chinese.
- Conflict Rate: Out of the 3,000 source questions, 73.9% (2,356 questions) were verified to be locale-sensitive, meaning they yielded conflicting, culturally dependent answers in at least one language.
Key Insights and Ideas
- The Translation Trap: Naive machine translation frequently causes semantic drift, particularly for culturally specific terms or ambiguous expressions. The authors' back-translation pipeline significantly reduced translation error rates, especially in languages like Arabic, Hebrew, and Chinese.
- Pretraining Bias Dictates Cultural Awareness: Models trained with a specific regional focus performed significantly better on conflict questions in their respective languages. For example, the EXAONE model excelled at Korean local context, while Qwen3 dominated in Chinese. This indicates that a model's cultural competence is directly tied to the geographic distribution of its pretraining data.
- Prompting Helps, But It's a Double-Edged Sword: Adding explicit locale guidance to prompts (e.g., telling the model to answer from the perspective of a specific region) improved performance across the board. However, the authors note that over-conditioning can sometimes cause models to output stereotype-driven responses.
Key Results
The authors evaluated five state-of-the-art models (GPT-4, Qwen3-14B, Gemma3-12B, LLaMA-3.1-8B, and EXAONE-7.8B):
- The English Gap: While the "Oracle LM" (GPT-4) maintained high accuracy across all languages (~88% Exact Match), smaller open-weights models showed a massive performance drop outside of English. For instance, LLaMA-3.1-8B achieved a 50.56 F1 score in English but plummeted to 16.92 F1 in Arabic.
- The Sensitivity Penalty: Performance dropped drastically when models moved from locale-invariant (non-conflict) questions to locale-sensitive (conflict) questions. Gemma3-12B’s average F1 score dropped from 56.96 on non-conflict questions to just 29.06 on conflict questions.
Limitations and Future Work
- Oracle Dependency: The pipeline relies heavily on proprietary models (GPT-4) for initial translation and answer generation, which inherently caps the diversity and quality of the generated data.
- Compute Constraints: Due to infrastructure limits, the researchers could not evaluate larger-scale open-source multilingual models.
- Stereotyping Risks: Forcing models to adapt to specific locales risks reinforcing cultural stereotypes or generating hallucinations in low-resource languages. The authors recommend incorporating human-in-the-loop validation when deploying these systems in high-stakes environments.
Summary updated