Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages

Alessio Buscemi, Cedric Lothritz, Sergio Morales, Marcos Gomez-Vazquez, Robert Clarisó, Jordi Cabot, German CastignaniarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) are known to inherit social biases from their training data, yet evaluation tools remain predominantly English-centric. This paper introduces MLA-BiTe (MultiLingual Augmented Bias Testing), a framework designed to automate the translation and paraphrasing of bias-testing templates. Our evaluation across six languages reveals a significant "language gap," where models exhibit higher and more volatile social biases in low-resource languages like Luxembourgish and Catalan compared to high-resource counterparts like English.

Introduction

As Large Language Models (LLMs) are integrated into global applications, the risk of propagating social biases increases. While benchmarks exist to detect these biases, they are largely limited to English. Manually adapting these benchmarks into hundreds of other languages is a resource-intensive process requiring native-level expertise. This creates a critical "blind spot" for low-resource or regionally co-official languages. A model that appears "safe" in English may still generate discriminatory content in languages with less training data representation. MLA-BiTe addresses this by leveraging LLMs to automate the creation of multilingual bias benchmarks while maintaining semantic integrity.

Methodology

The MLA-BiTe framework enhances the existing LangBiTe tool by automating two critical stages: translation and paraphrasing. The goal is to move test templates from a source language (English) to target languages and generate multiple variations to ensure bias detection is robust against specific wording choices.

The Augmentation Pipeline

The researchers explored two sequences: P2T (Paraphrase then Translate) and T2P (Translate then Paraphrase). While both yielded similar quality, the T2P pipeline was selected for the primary evaluation. GPT-4o served as the engine for this augmentation due to its superior instruction-following capabilities.

Algorithm 3: The paraphrasing procedure used to generate robust bias-testing variations within the MLA-BiTe framework.

Datasets and Language Selection

The study utilized templates from the LangBiTe repository, covering seven sensitive categories: Ageism, Lgbtiqphobia, Politics, Racism, Religion, Sexism, and Xenophobia. The evaluation spanned six languages across two families:

  • Romance: Spanish (High-resource), French (High-resource), and Catalan (Low-resource).
  • West Germanic: English (High-resource), German (High-resource), and Luxembourgish (Low-resource).

Key Insights

The Low-Resource Bias Penalty

Bias detection was most stable in English, where models showed the least discriminatory behavior. However, as the resource level of the language decreased, bias became more volatile. Luxembourgish exhibited the highest discrimination rates, suggesting that lower data volume in training sets correlates with higher manifested bias.

Linguistic Proximity vs. Bias Proximity

Interestingly, linguistic similarity did not guarantee similar bias profiles. While German and Luxembourgish are closely related, Luxembourgish biases aligned more closely with Spanish and Catalan. This indicates that bias is driven more by training data volume and specific cultural nuances than by genetic linguistic relationships.

The Challenge of Political Contexts

Political bias emerged as the most inconsistent category. The authors attribute this to the "American-centric" nature of the source templates. When models are prompted about U.S. political concepts in languages like Catalan, they often struggle to maintain the neutrality expected by the evaluation oracle.

Key Results

The study evaluated four state-of-the-art models: Claude 3.5 Sonnet, Gemini 1.5 Flash, Llama 3 405b, and GPT-4o.

  • Performance Leaders: GPT-4o and Claude 3.5 Sonnet achieved the highest "pass" rates for non-biased behavior, with scores averaging around 75%.
  • Bias Vulnerability: Llama 3 405b was identified as the most biased model in this framework, particularly in low-resource settings.
  • Process Reliability: The automated translation process proved highly effective. As shown in the results, models like Gemini 1.5 Flash and GPT-4o achieved 100% success in generating processable translations.
Success rates of automated translations across the evaluated LLMs, validating the reliability of the MLA-BiTe augmentation process.

Conclusion and Future Work

MLA-BiTe demonstrates that LLM-based augmentation is a viable path for scaling bias evaluations to low-resource languages, maintaining a high semantic overlap (0.85 to 0.95) with human-written templates. However, limitations remain regarding the "American-centricity" of current benchmarks. Future research will focus on developing "culture-aware" translation strategies to adapt prompts to local norms and extending the framework to multimodal models to investigate the language gap in visual outputs.

Summary updated

Abstract

Large Language Models (LLMs) have exhibited impressive natural language processing capabilities but often perpetuate social biases inherent in their training data. To address this, we introduce MultiLingual Augmented Bias Testing (MLA-BiTe), a framework that improves prior bias evaluation methods by enabling systematic multilingual bias testing. MLA-BiTe leverages automated translation and paraphrasing techniques to support comprehensive assessments across diverse linguistic settings. In this study, we evaluate the effectiveness of MLA-BiTe by testing four state-of-the-art LLMs in six languages -- including two low-resource languages -- focusing on seven sensitive categories of discrimination.
BibTeX
@misc{buscemi2025mind,
  author = {Alessio Buscemi and Cedric Lothritz and Sergio Morales and Marcos Gomez-Vazquez and Robert Clarisó and Jordi Cabot and German Castignani},
  title = {{Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages}},
  year = {2025},
  eprint = {2504.18560},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.18560},
}
Figure