Abstract
Large Language Models (LLMs) are known to inherit social biases from their training data, yet evaluation tools remain predominantly English-centric. This paper introduces MLA-BiTe (MultiLingual Augmented Bias Testing), a framework designed to automate the translation and paraphrasing of bias-testing templates. Our evaluation across six languages reveals a significant "language gap," where models exhibit higher and more volatile social biases in low-resource languages like Luxembourgish and Catalan compared to high-resource counterparts like English.
Introduction
As Large Language Models (LLMs) are integrated into global applications, the risk of propagating social biases increases. While benchmarks exist to detect these biases, they are largely limited to English. Manually adapting these benchmarks into hundreds of other languages is a resource-intensive process requiring native-level expertise. This creates a critical "blind spot" for low-resource or regionally co-official languages. A model that appears "safe" in English may still generate discriminatory content in languages with less training data representation. MLA-BiTe addresses this by leveraging LLMs to automate the creation of multilingual bias benchmarks while maintaining semantic integrity.
Methodology
The MLA-BiTe framework enhances the existing LangBiTe tool by automating two critical stages: translation and paraphrasing. The goal is to move test templates from a source language (English) to target languages and generate multiple variations to ensure bias detection is robust against specific wording choices.
The Augmentation Pipeline
The researchers explored two sequences: P2T (Paraphrase then Translate) and T2P (Translate then Paraphrase). While both yielded similar quality, the T2P pipeline was selected for the primary evaluation. GPT-4o served as the engine for this augmentation due to its superior instruction-following capabilities.
Datasets and Language Selection
The study utilized templates from the LangBiTe repository, covering seven sensitive categories: Ageism, Lgbtiqphobia, Politics, Racism, Religion, Sexism, and Xenophobia. The evaluation spanned six languages across two families:
- Romance: Spanish (High-resource), French (High-resource), and Catalan (Low-resource).
- West Germanic: English (High-resource), German (High-resource), and Luxembourgish (Low-resource).
Key Insights
The Low-Resource Bias Penalty
Bias detection was most stable in English, where models showed the least discriminatory behavior. However, as the resource level of the language decreased, bias became more volatile. Luxembourgish exhibited the highest discrimination rates, suggesting that lower data volume in training sets correlates with higher manifested bias.
Linguistic Proximity vs. Bias Proximity
Interestingly, linguistic similarity did not guarantee similar bias profiles. While German and Luxembourgish are closely related, Luxembourgish biases aligned more closely with Spanish and Catalan. This indicates that bias is driven more by training data volume and specific cultural nuances than by genetic linguistic relationships.
The Challenge of Political Contexts
Political bias emerged as the most inconsistent category. The authors attribute this to the "American-centric" nature of the source templates. When models are prompted about U.S. political concepts in languages like Catalan, they often struggle to maintain the neutrality expected by the evaluation oracle.
Key Results
The study evaluated four state-of-the-art models: Claude 3.5 Sonnet, Gemini 1.5 Flash, Llama 3 405b, and GPT-4o.
- Performance Leaders: GPT-4o and Claude 3.5 Sonnet achieved the highest "pass" rates for non-biased behavior, with scores averaging around 75%.
- Bias Vulnerability: Llama 3 405b was identified as the most biased model in this framework, particularly in low-resource settings.
- Process Reliability: The automated translation process proved highly effective. As shown in the results, models like Gemini 1.5 Flash and GPT-4o achieved 100% success in generating processable translations.
Conclusion and Future Work
MLA-BiTe demonstrates that LLM-based augmentation is a viable path for scaling bias evaluations to low-resource languages, maintaining a high semantic overlap (0.85 to 0.95) with human-written templates. However, limitations remain regarding the "American-centricity" of current benchmarks. Future research will focus on developing "culture-aware" translation strategies to adapt prompts to local norms and extending the framework to multimodal models to investigate the language gap in visual outputs.
Summary updated