Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models

Introduction

Large Language Models (LLMs) are increasingly deployed globally, yet their safety mechanisms often remain "English-centric." This research highlights a critical phenomenon: LLMs appear to possess different "moral compasses" depending on the language of the prompt. Even when questions are semantically identical, a model might provide a nuanced response in English while refusing to answer or providing unsafe content in another language. This "moral schizophrenia" poses significant risks for the equitable and safe deployment of AI worldwide.

Methodology

To move beyond anecdotal evidence, the researchers developed a rigorous pipeline to statistically prove cross-language ethical inconsistency. The process begins with crawling news from Human Rights Watch across 17 sensitive topics, such as Women’s Rights and Torture. These articles serve as the basis for generating controversial questions using GPT-4, which are then translated into six target languages: English, Korean, Chinese, Spanish, German, and Hindi.

Expand

Source, p. 3

The core of the methodology lies in its "statistical teeth." The researchers employed McNemar’s Test to determine if the probability of a model rejecting a question differs significantly between language pairs. Additionally, PERMANOVA (Permutational Multivariate Analysis of Variance) was used on text embeddings to analyze whether the actual content and distribution of "acceptable" responses vary by language.

The MSQAD Dataset

The study introduces the Multilingual Sensitive Questions & Answers Dataset (MSQAD). Unlike existing benchmarks that often rely on Western-centric stereotypes, MSQAD is grounded in real-world global news. It encompasses thousands of socially sensitive questions and paired responses across 17 global topics and 6 languages, providing a robust foundation for evaluating international human rights contexts in AI.

Results and Analysis

The results consistently rejected the null hypothesis, indicating that ethical differences across languages are statistically significant biases rather than random noise.

• The Refusal Gap: Chinese and Hindi prompts resulted in significantly higher refusal rates compared to English. Conversely, Spanish and German prompts were more likely to trigger responses that violated safety guidelines which the English version would have blocked.
• English as the Baseline: English responses were consistently the most detailed and informative. In other languages, "ethical informativeness" dropped, suggesting that safety tuning is most effective in English and does not fully transfer across the model's multilingual capabilities.
• Statistical Significance: At a 5% significance level, McNemar’s Test rejected the null hypothesis (that rejection rates are equal across languages) in 94.08% of tested cases.

Expand

Source, p. 5

Furthermore, PERMANOVA results showed that the embeddings of English responses were significantly more distinct and detailed than those of other languages, highlighting a clear disparity in the quality of information users receive based on their language.

Conclusion and Future Work

This research proves that model evolution does not inherently fix bias; in fact, newer models like Llama-3 often show more pronounced language-based ethical disparities. While the study is limited to six major languages and relies on a GPT-4-generated dataset, it sets a new standard for multilingual safety evaluation. Future work will aim to expand linguistic diversity and investigate the impact of these biases on downstream tasks like automated moderation and legal advice.

@inproceedings{yu2025delving,
  author = {Seunguk Yu and Juhwan Choi and YoungBin Kim},
  title = {{Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}