Most evaluations of Large Language Models (LLMs) rely on a "translate-test" paradigm: taking a successful English benchmark, like MMLU, and machine-translating it into other languages. However, as Pikuliak et al. argue, this approach often fails to capture the unique linguistic nuances and cultural expressions of native languages. A growing body of research suggests that to truly measure a model's regional proficiency, we must move toward native-source benchmark construction—building evaluation suites from the ground up using materials born within the target culture.
The Failure of Translation
Translation does more than just swap words; it often imports the cultural logic, legal frameworks, and social norms of the source language (usually English). When a benchmark is translated, it carries "translationese" artifacts—syntactic awkwardness that native speakers never actually produce. Hu 0001 demonstrates that by eliciting hypotheses directly from native speakers rather than translating them, researchers can create benchmarks that reflect natural phrasing and culturally relevant contexts, exposing a significant gap between AI and human performance that translated tests hide.
Furthermore, translation often misses the "concept shift" inherent in how different societies describe the world. Kabra et al. found that the primary barrier to AI understanding figurative language isn't the vocabulary, but the deep-seated cultural differences in metaphors. Similarly, Alghamdi et al. show that native-authored math word problems capture subtle mathematical cues and cultural relevance that machine translation simply erases.
Sourcing from Local Realities
The most effective way to ground a model in a specific region is to source evaluation data from high-stakes, real-world materials already used by that society. Several researchers have turned to national standardized exams:
- Academic Excellence: Koto et al. and Al-Ballaa et al. developed Arabic benchmarks by sourcing questions from local school curricula and the Saudi General Aptitude Test. These reveal that even top-tier models like GPT-4 struggle with regional nuances compared to Western-centric data.
- Professional Standards: Gili et al. and the authors of ITALIC utilized Italian public sector employment and university entrance exams. This ensures the evaluation captures local legal and administrative frameworks that a translated English test would never include.
- Linguistic Fidelity: For morphologically rich languages like Turkish, Yüksel et al. show that using questions from local curriculum experts avoids the errors and biases common in machine-translated datasets.
Capturing Cultural Common Sense
"Commonsense" is not universal; it is geographically and culturally contingent. Yin et al. introduced benchmarks where native annotators define ground-truth facts based on local reality, revealing that a model's size doesn't necessarily correlate with its cultural awareness.
To capture this "cultural common sense," Nayak et al. recruited long-term residents to generate questions that are intuitive to locals but difficult for outsiders. This moves evaluation beyond simple object recognition toward interpreting the deeper meaning of rituals or garments. Similarly, Mohamed demonstrated that emotional responses to art are deeply tied to cultural backgrounds, requiring native-speaker annotations to capture subjective nuances lost in translation.
The Performance Gap
When models are tested on these native-source benchmarks, a "cultural gap" emerges. Verma et al. found that while global models perform well in STEM subjects across Indian languages, they often perform no better than random chance on India-specific knowledge like local laws and history.
By prioritizing native-first data—whether through human curation of local news Leidinger, sourcing organic multilingual media Sanders et al., or using local knowledge seeds to generate questions Sakai et al.—researchers are building a more honest mirror for AI. As Kim et al. suggest, bridging this gap is essential for moving LLMs from academic curiosities to reliable tools in real-world, regional applications.