With Prejudice to None: A Few-Shot, Multilingual Transfer Learning Approach to Detect Social Bias in Low Resource Languages

Nihar Sahoo, Niteesh Mallela, Pushpak BhattacharyyaACL 2023 · DOI

Introduction

While AI safety research has made significant strides in identifying social bias, the vast majority of this work is centered on high-resource languages like English. However, social bias is not universal; it is deeply embedded in specific cultural, historical, and linguistic contexts. A model optimized for Western contexts may fail to recognize a caste-based slur in Hindi or regional prejudices in Italian.

This research addresses two primary challenges: the "data drought" for social bias in low-resource languages and the inadequacy of simple machine translation. Translation often sanitizes or misinterprets the nuances of cultural prejudice, making it an unreliable filter for bias detection.

A New Benchmark for Hindi Social Bias

To bridge this gap, the researchers introduced a novel dataset of approximately 9,000 Hindi social media posts sourced from Instagram, Facebook, and Twitter. Unlike Western-centric datasets, this collection captures region-specific biases such as Caste Bias and Personal Favor (favoritism toward political or celebrity figures).

Examples of annotated Hindi social media posts showing bias categories, target groups, and rationales.

Each post is meticulously annotated for bias category, target group, sentiment, and a human-written rationale. These rationales are crucial, as they provide the underlying logic for why a post is considered biased, moving beyond simple keyword matching.

Methodology: Sequential Few-Shot Transfer Learning

The core of the proposed framework is Sequential Fine-tuning using XLM-RoBERTa (XLM-R), a transformer model pre-trained on 100 languages. The process involves two stages:

  1. Source Training: The model is first fine-tuned on a large, high-resource English dataset (Stereoset).
  2. Target Fine-tuning: The model is then fine-tuned on a small "few-shot" sample of the target language (Hindi, Italian, or Korean).

This approach tests the hypothesis that the concept of biased reasoning—such as stereotyping—transfers across languages, even when the specific targets and linguistic expressions change.

Key Results and Insights

The study yielded several critical findings regarding the efficiency of multilingual models:

  • The Failure of Translation: Detecting bias in translated text (Hindi to English) resulted in a significantly lower F1 score (56.6) compared to native detection (80.8). This confirms that linguistic nuances are essential for identifying prejudice.
  • Data Efficiency: One of the most striking results is the "sweet spot" of few-shot learning. By starting with an English-trained model, researchers found that only 150 to 350 examples in the target language were needed to achieve performance levels comparable to models trained on thousands of native examples.
Macro F1 scores across Hindi, Korean, and Italian as a function of the number of few-shot examples (N).

As shown in the performance curves, the sequential fine-tuning approach (XLM_ENG_L) consistently outperforms standard multilingual fine-tuning (XLM_L), especially in low-data regimes. In Hindi, the macro F1 score jumped from 59.2 in a zero-shot setting to over 80.0 with minimal target-language data.

Conclusion

This work demonstrates that while social bias is culturally specific, the underlying patterns of biased reasoning can be effectively transferred across languages. By leveraging sequential few-shot learning, robust bias detectors can be built for low-resource languages without the need for massive, expensive datasets.

Future research will focus on incorporating broader context—such as user history and conversation threads—to identify more subtle "dog-whistle" biases and exploring multi-task learning to simultaneously predict bias, sentiment, and rationales.

Summary updated

BibTeX
@inproceedings{sahoo2023with,
  author = {Nihar Sahoo and Niteesh Mallela and Pushpak Bhattacharyya},
  title = {{With Prejudice to None: A Few-Shot, Multilingual Transfer Learning Approach to Detect Social Bias in Low Resource Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.842},
}
Figure