Introduction
Large language models (LLMs) are increasingly integrated into global communication systems, yet research into their inherent biases remains heavily focused on English. This paper investigates how the sophisticated politeness systems of Japanese and Korean—languages that encode social hierarchy through honorifics and verb endings—serve as proxies for gender bias. The researchers demonstrate that models internalize cultural stereotypes, associating specific speech levels with gender, which can be exploited as an "attack vector" to bypass safety filters like cyberbullying detectors.
Problem Statement: Beyond English-Centric Bias
Traditional NLP bias research focuses on pronouns or occupational stereotypes. However, in Japanese and Korean, social information is embedded in politeness levels. Sociolinguistic norms often dictate that women use more polite speech, while men may use ruder or more formal styles. The authors address a critical gap: Do LLMs internalize these social prejudices? If a model equates "polite" with "female" and "rude" with "male," this representational bias can lead to allocational bias—for instance, a toxic comment might evade a cyberbullying filter simply by being phrased politely.
Methodology
The study employed a multi-faceted approach to probe and mitigate these biases:
- Template-based Probing: Using masked language modeling, the researchers measured whether models predicted "he" or "she" more frequently when verbs were conjugated into different politeness levels (from rude to highly formal).
- Contextual Analysis: Physical locations (e.g., "Parliament" vs. "Daycare") were added to templates to observe the influence of environmental context on gender prediction.
- Attack Vector Simulation: A Japanese cyberbullying detection model was "attacked" by modifying the politeness levels of toxic tweets and appending gender terms to see if the filter could be bypassed.
- Mitigation: The researchers utilized SetFit (Sentence Transformer Fine-Tuning), a few-shot contrastive learning framework, to develop a more robust model capable of identifying toxic content regardless of politeness style.
Key Insights: Politeness as a Gender Proxy
The analysis revealed that models do not merely learn grammar; they absorb social prejudice. In Japanese, "informal polite" speech (teineigo) was strongly linked to female identities, while "rude" and "formal" speech were associated with males. In Korean, the use of honorifics was the strongest predictor of a "male" classification, suggesting models equate being a subject of respect with masculinity.
This "Honorific Shield" allows politeness to cloak toxicity. A baseline detection model's performance dropped significantly when toxic tweets used honorifics, particularly when targeting women. This highlights the danger of translating English bias benchmarks without accounting for local linguistic nuances.
Results and Mitigation
The study found a prevalent male bias across nearly all 20 tested models (10 Japanese, 10 Korean), where "he" was predicted significantly more often than "she" regardless of politeness.
A popular Japanese cyberbullying model saw its F1 score of 0.40 plummet when faced with honorific-based attacks. However, the proposed SentenceBERT model, trained via few-shot learning on a specialized attack dataset, achieved a robust F1 score of 0.82, maintaining high performance across all genders and politeness levels.
Conclusion and Future Work
While this study highlights critical vulnerabilities in non-English LLMs, it is limited to common verbs and basic politeness levels. Real-world usage involves more complex nuances, such as specialized nouns and titles. Future research should investigate whether these biases persist in larger generative models like GPT-4. By exposing these vulnerabilities, the authors aim to pave the way for fairer and more robust NLP systems globally.
Summary updated