KpopMT: Translation Dataset with Terminology for Kpop Fandom

JiWoo Kim, Yunsu Kim, JinYeong Bak

Abstract

While machines learn from existing corpora, humans have the unique capability to establish and accept new language systems. This makes humans form unique language systems within social groups. Aligning with this, we focus on a gap remaining in addressing translation challenges within social groups, where in-group members utilize unique terminologies. We propose the KpopMT dataset, which aims to fill this gap by enabling precise terminology translation, choosing Kpop fandom as an initiative for social groups given its global popularity. Expert translators provide 1,000 English translations for Korean posts and comments, each annotated with specific terminology within social groups' language systems. We evaluate existing translation systems, including GPT models, on KpopMT to identify their failure cases. Results show overall low scores, underscoring the challenges of reflecting group-specific terminologies and styles in translation. We make KpopMT publicly available.

Introduction: The Terminology Gap in Social Dialects

Machine translation (MT) has reached high proficiency in standard languages but often falters when encountering "social dialects"—the specialized linguistic systems and jargon developed within specific online communities. Within the global Kpop fandom, terms like "stan," "bias," and "moot" carry specific social weight that general-purpose translators often fail to capture.

Comparison of Korean and English Kpop fandom language systems.

The authors identify a "terminology gap": in-group members use words that have specific meanings within their community which general-purpose translators often render literally or incorrectly. For example, translating a post about "stanning a bias" as "focusing on a favorite" misses the community-specific identity. KpopMT provides a benchmark for translating these culturally rich, niche language systems.

Methodology: Building KpopMT

The researchers developed a multi-phase pipeline to create a high-quality, terminology-tagged dataset:

  1. Data Collection: Korean posts and comments were scraped from fan community sites (Theqoo, Instiz) and Twitter (X) using a query list of fandom terms.
  2. Expert Translation: Ten translators fluent in Kpop jargon produced 1,000 English translation pairs. Translators were specifically instructed to use English fandom equivalents (like "stan" or "pc" for photo card) rather than standard English.
  3. Terminology Tagging: The dataset includes a "termbase" (glossary) where terms are categorized into three types: Group-Lexicon (unique fandom vocabulary), Group-NE (named entities like idol nicknames), and Slang (general internet variations).
The KpopMT dataset creation pipeline, showing sentence translation and terminology tagging phases.

Key Insights and Results

Evaluation of several models, including GPT-3.5, GPT-4, and Google Translate, revealed significant challenges in handling social dialects:

  • GPT-4 as Terminology Leader: GPT-4 achieved the highest Exact-Match term Accuracy (EMA) of 26.4, significantly outperforming Google Translate (10.4). It excelled at common terms like "bias" and "stan."
  • The Fluency vs. Jargon Trade-off: While Google Translate maintained the highest BLEU score (13.9), it often missed community-specific jargon. This indicates that models optimized for standard fluency may struggle with social authenticity.
  • Social Connection Priority: A human survey of native English-speaking Kpop fans found that 89.75% preferred translations using fandom terms over standard language. Using the "right" jargon makes fans feel more connected to the community.
  • Overall Difficulty: The fact that the top BLEU score was only 13.9 (compared to much higher scores on standard datasets) underscores just how difficult social dialects are for current AI.

Conclusion and Future Work

The KpopMT dataset highlights that "Kpop fandom" is not a monolith, as different groups have their own distinct sub-dialects. Future work aims to expand KpopMT to include other social groups, such as sports fans and global movie communities, to further refine how MT handles the diverse ways humans build identity through language.

Summary updated

Abstract

While machines learn from existing corpora, humans have the unique capability to establish and accept new language systems.This makes human form unique language systems within social groups.Aligning with this, we focus on a gap remaining in addressing translation challenges within social groups, where in-group members utilize unique terminologies.We propose KpopMT dataset, which aims to fill this gap by enabling precise terminology translation, choosing Kpop fandom as an initiative for social groups given its global popularity.Expert translators provide 1k English translations for Korean posts and comments, each annotated with specific terminology within social groups' language systems.We evaluate existing translation systems including GPT models on KpopMT to identify their failure cases.Results show overall low scores, underscoring the challenges of reflecting group-specific terminologies and styles in translation.We make KpopMT publicly available.
BibTeX
@misc{kim2024kpopmt,
  author = {JiWoo Kim and Yunsu Kim and JinYeong Bak},
  title = {{KpopMT: Translation Dataset with Terminology for Kpop Fandom}},
  year = {2024},
  eprint = {2407.07413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.07413},
}
Figure