TL;DR
This paper introduces the Korean Stereotype Content Model (KoSCM), extending the established Stereotype Content Model (SCM) to the Korean language. By creating a Korean lexicon of warmth and competence seed words and a corresponding dataset, the authors demonstrate the SCM's cross-cultural applicability and build a computational model for analyzing Korean stereotypes.
What Problem Does This Paper Address?
As AI systems become more integrated into our lives, understanding and mitigating biases within them is crucial. Language models, in particular, can perpetuate harmful stereotypes present in their training data. While the Stereotype Content Model (SCM) has been a valuable framework for analyzing stereotypes in English-speaking cultures, its cross-cultural validity and applicability to other languages, like Korean, remain underexplored. This paper addresses this gap by adapting the SCM for Korean, aiming to broaden our understanding of how stereotypes manifest computationally across different linguistic and cultural contexts.
Methods
The authors propose a four-step framework to translate the SCM into Korean and create a new dataset:
- Extract Seed Words: They begin by identifying English seed words associated with warmth and competence from existing lexicons.
- Translate Seed Words: These English seed words are translated into Korean using a machine translation model (Naver Papago) and then validated by an expert translator to ensure accuracy and prevent redundancy. This process results in 285 validated Korean seed words.
- Generate Sentences: Using curated templates designed to avoid referencing specific social groups, sentences are generated by inserting the Korean seed words. For example, "That person is a [seed word] person."
- Augment Data with Back-Translation: To address potential low-resource challenges and increase data diversity, sentences are back-translated from Korean to English and then back to Korean using the NLLB model. This augmentation yields a dataset of 3,420 sentences.
Finally, they use this dataset to train two types of SCM models: one based on embedding models like mBERT, and another using in-context learning with large language models (LLMs) like Korean Llama (kLlama).
Datasets
The primary dataset introduced in this paper is the KoSCM dataset, comprising 3,420 Korean sentences labeled for warmth and competence. This dataset was created using the translation and augmentation methods described above. For evaluation, they also utilize the Korean Offensive Language Dataset (KOLD), which contains comments labeled with group information relevant to Korean culture.
Key Insights and Ideas
- SCM's Cross-Cultural Potential: The study provides evidence that the SCM framework, a cornerstone of social psychology research on stereotypes, can be computationally adapted and applied to a non-Western language like Korean. This suggests that the underlying dimensions of warmth and competence might be more universal than previously assumed in computational linguistics.
- The Importance of Lexicon Translation and Validation: Simply translating lexicons isn't enough. The rigorous validation by an expert translator was crucial to ensure the accuracy and distinctiveness of the Korean seed words, highlighting the need for careful linguistic and cultural adaptation.
- LLM Performance Gap: The research reveals a significant performance gap between traditional embedding-based models (like mBERT) and LLMs (like kLlama) when using in-context learning for stereotype prediction. While mBERT achieved high accuracies, kLlama's performance was considerably lower, potentially due to data limitations and the nuances of in-context learning.
- Outliers and Data Scarcity: The analysis of stereotypes for various social groups in Korean revealed some discrepancies with previous survey findings. The authors attribute these outliers, such as "public functionaries," to insufficient data for those specific groups, underscoring the challenge of data scarcity in cross-cultural NLP research.
Key Results
- The mBERT model achieved high prediction accuracies for both warmth (0.9230) and competence (0.9376) on the KoSCM dataset.
- The LLM-based models (kLlama) showed significantly lower performance, with accuracies around 0.5, even when using carefully selected samples for few-shot learning.
- The analysis of social groups using the best-performing model (mBERT) showed a general alignment with the SCM's hypothesized clusters (e.g., low-competence/high-warmth for "women" and "elderly").
- A positive correlation (0.71) was found between perceived status (proxied by average wage) and competence in Korean, supporting the social structural correlates hypothesis of the SCM. This correlation is comparable to previous findings in South Korea (0.64) and universal averages (0.79).
Limitations and Future Work
The authors acknowledge several limitations:
- Authorial Bias: Bias can influence experimental design and analysis, particularly in prompt curation and hyperparameter selection for clustering.
- Scale Constraints: The lack of extensive open-source Korean datasets and models limits the performance of the models compared to their English counterparts. Noise was observed in back-translated data, likely due to these resource limitations.
- Framework Specificity: The KoSCM dataset and analysis are based on the specific SCM framework, meaning they don't encompass all possible perspectives on stereotypes.
Future work includes:
- Expanding the research to incorporate additional languages.
- Developing algorithms that can guide and transform stereotypes present in sentences.
- Exploring alternative methodologies to gain a more comprehensive understanding of stereotypes.
Summary updated