KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Application

Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Gunhee Kim, Jung-Woo HaarXiv 2023 · OpenReview

Abstract

Large language models (LLMs) learn not only natural text generation abilities but also social biases against different demographic groups from real-world data. This poses a critical risk when deploying LLM-based applications. Existing research and resources are not readily applicable in South Korea due to the differences in language and culture, both of which significantly affect the biases and targeted demographic groups. This limitation requires localized social bias datasets to ensure the safe and effective deployment of LLMs. To this end, we present KO SB I, a new social bias dataset of 34k pairs of contexts and sentences in Korean covering 72 demographic groups in 15 categories. We find that through filtering-based moderation, social biases in generated content can be reduced by 16.47%p on average for HyperCLOVA (30B and 82B), and GPT-3.

Introduction: The Need for Localized Bias Mitigation

While Large Language Models (LLMs) demonstrate remarkable capabilities in generating human-like text, they often act as "stochastic sponges," absorbing and propagating toxic social biases present in their training data. Most existing research on bias mitigation focuses on English-centric and Western cultural norms. However, social bias is deeply intertwined with language and culture. In South Korea, specific biases regarding regionalism ("domestic area of origin"), marital status, and socio-economic hierarchies (e.g., the "silver spoon" concept) are prevalent but often overlooked in Western datasets.

Existing Korean datasets primarily target explicit hate speech or profanity, leaving a significant gap in addressing subtle, implicit stereotypes. To address this, we introduce the Korean Social Bias (KoSBi) dataset, designed to identify and mitigate these localized risks.

Methodology

The KoSBi dataset was constructed using a multi-step pipeline:

  1. Data Generation: To ensure balanced representation for minority groups, we utilized HyperCLOVA to generate context-sentence pairs via in-context few-shot learning.
  2. Human-in-the-Loop Annotation: 200 crowd workers reviewed the generated pairs, labeling them as "safe" or "unsafe." Unsafe samples were further categorized into stereotypes (cognitive), prejudice (emotional), or discrimination (behavioral).
  3. Ambiguity Filtering: To create a robust benchmark, we specifically selected ambiguous samples that are challenging for safety classifiers to distinguish.
  4. Mitigation Strategy: We employed a filtering-based moderation approach where a "safe sentence classifier" trained on KoSBi selects the most appropriate response from multiple LLM-generated candidates.

Dataset Characteristics

KoSBi consists of 34,214 context-sentence pairs covering 72 demographic groups across 15 categories. Unlike previous benchmarks, KoSBi emphasizes the importance of context; a neutral sentence can become unsafe depending on the preceding statement.

Comparison of KoSBi with existing Korean toxicity datasets.

As shown in the comparison above, KoSBi is significantly more comprehensive than existing Korean datasets like BEEP! or APEACH, particularly in its demographic coverage and the inclusion of context, which is vital for identifying implicit bias.

Experimental Results

Our experiments demonstrate that KoSBi-based filtering significantly enhances the safety of LLM outputs. A classifier trained on KoSBi achieved a 71.21% Macro F1 score, outperforming models trained on less comprehensive datasets.

Safety performance of LLMs with and without KoSBi-based filtering.

Across models including GPT-3 and HyperCLOVA (30B and 82B), the application of KoSBi-based filtering reduced unsafe generations by an average of 16.47 percentage points. For HyperCLOVA (82B), safe responses increased by 15.3%p, while GPT-3 saw an increase of 15.6%p. While filtering improves safety, we observed a slight trade-off in conversational coherency, as the safest response may occasionally be less relevant than a riskier alternative.

Conclusion

The KoSBi dataset provides a critical resource for developing safer LLMs in the Korean context. By targeting implicit social biases and providing a large-scale, context-aware dataset, we enable more effective moderation strategies. Future work will focus on balancing the safety-utility trade-off and expanding the dataset to include even more diverse social groups and complex conversational structures.

Summary updated

Abstract

Large language models (LLMs) learn not only natural text generation abilities but also social biases against different demographic groups from real-world data. This poses a critical risk when deploying LLM-based applications. Existing research and resources are not readily applicable in South Korea due to the differences in language and culture, both of which significantly affect the biases and targeted demographic groups. This limitation requires localized social bias datasets to ensure the safe and effective deployment of LLMs. To this end, we present KO SB I, a new social bias dataset of 34k pairs of contexts and sentences in Korean covering 72 demographic groups in 15 categories. We find that through filtering-based moderation, social biases in generated content can be reduced by 16.47%p on average for HyperCLOVA (30B and 82B), and GPT-3.
BibTeX
@misc{lee2023kosbi,
  author = {Hwaran Lee and Seokhee Hong and Joonsuk Park and Takyoung Kim and Gunhee Kim and Jung-Woo Ha},
  title = {{KoSBi: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Application}},
  year = {2023},
  eprint = {2305.17701},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.17701},
}
Figure