Introduction
Algorithmic bias in Natural Language Processing (NLP) has predominantly been studied within the context of the English language and United States socio-cultural norms. However, stereotypes are often culturally specific; a bias prevalent in one region may be irrelevant in another. Furthermore, linguistic structures, such as grammatical gender in French, present unique challenges for standard bias measurement techniques like "minimal pair" testing.
To address these issues, researchers have developed French CrowS-pairs, a benchmark adapted from the original English CrowS-pairs dataset. This work involves translating existing pairs, crowdsourcing new French-specific stereotypes, and evaluating the social biases present in French masked language models (MLMs).
Methods and Dataset Development
The creation of the French CrowS-pairs dataset involved a multi-step process to ensure cultural and linguistic relevance:
- Translation and Adaptation: 1,467 sentence pairs were adapted from the original English dataset. Rather than literal translation, the authors modified names and cultural references to maintain the stereotype's impact in a French context.
- Structural Refinement: The team corrected existing flaws in the original dataset, such as "double switches" (where multiple word changes obscure the bias signal) and "bias mismatches."
- Crowdsourcing Native Stereotypes: Using the LanguageARC platform, 210 new sentence pairs were collected natively in French. These capture local nuances, such as biases against specific neighborhoods (e.g., the 93 department) or regional accents.
- Evaluation Framework: Four models were tested: CamemBERT, FlauBERT, FrALBERT (French-specific), and mBERT (multilingual). The primary metric was the "stereo score," representing the frequency with which a model prefers a stereotypical sentence over a non-stereotypical counterpart.
Key Insights: Linguistic and Cultural Nuance
The adaptation process highlighted significant hurdles in cross-lingual bias benchmarking:
- Grammatical Gender Constraints: In English, swapping "he" for "she" rarely requires further sentence modification. In French, gendered nouns often require agreement from adjectives and verbs. Maintaining "minimal pairs" (differing by only one token) required careful paraphrasing.
- Cultural Non-Equivalence: Many US-centric tropes, such as police officers and donuts, do not translate to French culture. Conversely, French-specific biases regarding regional identities (e.g., Brittany) are absent from English datasets. This underscores the necessity of locally-grounded benchmarks.
Experimental Results
Evaluation across various models revealed that all tested French MLMs exhibit significant social bias, with scores consistently exceeding the neutral threshold of 50.0.
Key findings include:
- CamemBERT exhibited the highest overall bias on the French dataset with a metric score of 59.3.
- FrALBERT showed particularly high bias in the Religion category, reaching a score of 72.2.
- mBERT demonstrated the lowest bias on the French dataset (50.9) but was significantly less effective on English benchmarks compared to native English models.
- Category-Specific Bias: High bias scores were frequently observed in categories related to Socioeconomic Status and Nationality, including the newly added French-specific regional stereotypes.
Conclusion and Future Directions
The French CrowS-pairs dataset demonstrates that social bias is a persistent feature of French language models, deeply intertwined with both linguistic structure and cultural context. While this work bridges a significant gap, the authors note that the scope remains limited to Western cultures. Future research should expand these methodologies to non-Western languages and explore how these biases manifest in large-scale generative models.
Summary updated