XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering

Keon-Woo Roh, Yeong-Joon Ju, Seong-Whan Lee

TL;DR

Current multilingual QA benchmarks assume that translating a question shouldn't change its answer. This paper introduces XLQA, a new benchmark that explicitly accounts for cultural and regional variations across eight languages, revealing that state-of-the-art LLMs frequently fail when answers depend on local context.

What Problem Does This Paper Address?

Most multilingual open-domain question answering (ODQA) benchmarks rely on a flawed assumption: locale invariance. They assume that if you translate a question like "How long did it take the Twin Towers to be built?" into Korean or Chinese, the answer should still refer to the World Trade Center in New York.

In reality, a speaker in Seoul asking about the "Twin Towers" is likely referring to the LG Twin Towers, while a speaker in China might mean the Tianjin IFC.

Figure 1: Knowledge conflict in multilingual ODQA. Although all versions of the question aim to ask how long it took to build the "Twin Towers", different lan- guages elicit different answers based on locale-variant understanding. While English and Arabic refer to the World Trade Center (11 years), Korean and Chinese interpret "Twin Towers" as the LG Twin Towers and Tianjin IFC, respectively.

By ignoring these regional and cultural nuances, current evaluation metrics suffer from a severe bias—they penalize language models for providing answers that are locally accurate but differ from the original English ground truth.

Methods

To build a benchmark capable of evaluating this "locale awareness" without requiring prohibitive amounts of manual labor, the authors developed a scalable, three-stage pipeline:

  1. Multilingual Question Generation: English seed questions were translated into seven target languages using GPT-4. To eliminate translation artifacts and "semantic drift" (subtle shifts in meaning), they implemented a back-translation filtering step. Questions were translated back into English, and an LLM-as-judge evaluated whether the original and back-translated questions were semantically identical.
  2. Locale-Aware Answer Generation: The translated questions were fed into an LLM connected to a retrieval-augmented generation (RAG) web search component. The model was prompted to find answers grounded in language-specific sources (like localized Wikipedia). If the generated answer diverged semantically from the English baseline, it was flagged as a potential "knowledge conflict."
  3. Human Verification: Human annotators (via Amazon Mechanical Turk) reviewed the flagged questions, answers, and supporting evidence using a majority-voting scheme to ensure absolute correctness and verify whether the question was truly locale-sensitive.
Figure 2: The overall pipeline for constructing the XLQA benchmark. The process consists of three stages: (1) Multilingual Question Generation generates multilingual questions based on seed questions from existing QA datasets. (2) Locale-Aware Answer Generation uses LLM to generate locale-aware answers. (3) Human Verification verifies the answers with supporting evidence. The Output is a high-quality, locale-aware multilingual QA dataset.

Datasets

  • Seed Data: 3,000 high-quality English questions filtered from existing benchmarks (MKQA, MLQA, and HotpotQA).
  • XLQA (New Benchmark): A dataset of 24,000 question-answer-evidence triples spanning 8 languages: English, Korean, Arabic, Hebrew, Japanese, Russian, Vietnamese, and Simplified Chinese.
  • Conflict Rate: Out of the 3,000 source questions, 73.9% (2,356 questions) were verified to be locale-sensitive, meaning they yielded conflicting, culturally dependent answers in at least one language.

Key Insights and Ideas

  • The Translation Trap: Naive machine translation frequently causes semantic drift, particularly for culturally specific terms or ambiguous expressions. The authors' back-translation pipeline significantly reduced translation error rates, especially in languages like Arabic, Hebrew, and Chinese.
  • Pretraining Bias Dictates Cultural Awareness: Models trained with a specific regional focus performed significantly better on conflict questions in their respective languages. For example, the EXAONE model excelled at Korean local context, while Qwen3 dominated in Chinese. This indicates that a model's cultural competence is directly tied to the geographic distribution of its pretraining data.
  • Prompting Helps, But It's a Double-Edged Sword: Adding explicit locale guidance to prompts (e.g., telling the model to answer from the perspective of a specific region) improved performance across the board. However, the authors note that over-conditioning can sometimes cause models to output stereotype-driven responses.

Key Results

The authors evaluated five state-of-the-art models (GPT-4, Qwen3-14B, Gemma3-12B, LLaMA-3.1-8B, and EXAONE-7.8B):

  • The English Gap: While the "Oracle LM" (GPT-4) maintained high accuracy across all languages (~88% Exact Match), smaller open-weights models showed a massive performance drop outside of English. For instance, LLaMA-3.1-8B achieved a 50.56 F1 score in English but plummeted to 16.92 F1 in Arabic.
  • The Sensitivity Penalty: Performance dropped drastically when models moved from locale-invariant (non-conflict) questions to locale-sensitive (conflict) questions. Gemma3-12B’s average F1 score dropped from 56.96 on non-conflict questions to just 29.06 on conflict questions.

Limitations and Future Work

  • Oracle Dependency: The pipeline relies heavily on proprietary models (GPT-4) for initial translation and answer generation, which inherently caps the diversity and quality of the generated data.
  • Compute Constraints: Due to infrastructure limits, the researchers could not evaluate larger-scale open-source multilingual models.
  • Stereotyping Risks: Forcing models to adapt to specific locales risks reinforcing cultural stereotypes or generating hallucinations in low-resource languages. The authors recommend incorporating human-in-the-loop validation when deploying these systems in high-stakes environments.

Summary updated

Abstract

Large Language Models (LLMs) have shown significant progress in Open-domain question answering (ODQA), yet most evaluations focus on English and assume locale-invariant answers across languages. This assumption neglects the cultural and regional variations that affect question understanding and answer, leading to biased evaluation in multilingual benchmarks. To address these limitations, we introduce XLQA, a novel benchmark explicitly designed for locale-sensitive multilingual ODQA. XLQA contains 3,000 English seed questions expanded to eight languages, with careful filtering for semantic consistency and human-verified annotations distinguishing locale-invariant and locale-sensitive cases. Our evaluation of five state-of-the-art multilingual LLMs reveals notable failures on locale-sensitive questions, exposing gaps between English and other languages due to a lack of locale-grounding knowledge. We provide a systematic framework and scalable methodology for assessing multilingual QA under diverse cultural contexts, offering a critical resource to advance the real-world applicability of multilingual ODQA systems. Our findings suggest that disparities in training data distribution contribute to differences in both linguistic competence and locale-awareness across models.
BibTeX
@inproceedings{roh2025xlqa,
  author = {Keon-Woo Roh and Yeong-Joon Ju and Seong-Whan Lee},
  title = {{XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure