KpopMT: Translation Dataset with Terminology for Kpop Fandom

JiWoo Kim, Yunsu Kim, JinYeong Bak

Abstract

KpopMT is a specialized translation dataset designed to bridge the gap between standard machine translation and the nuanced "social dialects" used by global fandoms. By providing 1,000 Korean-English sentence pairs annotated with K-pop-specific jargon, this research demonstrates that even state-of-the-art models like GPT-4 struggle to accurately translate the unique terminology that defines online communities.

The Challenge of Social Dialects

Machine translation (MT) has achieved high performance in general and technical domains, yet it remains remarkably poor at handling social dialects—the unique linguistic systems, slang, and terminologies that emerge within specific social groups.

Examples of K-pop fandom terminology in Korean and English, highlighting terms like 'bias' and 'stan'.

In the K-pop fandom, specialized language (e.g., "stan," "bias," or "poca") is essential for community building. Standard MT systems often "correct" these terms into standard language or mistranslate them entirely because they are underrepresented in general training corpora. For instance, a fan's reference to their "bias" might be translated as "favorites," stripping the term of its specific cultural weight.

Methodology: The KpopMT Dataset

The researchers developed the KpopMT dataset using a rigorous two-phase approach to ensure both translation quality and terminological accuracy.

Overview of the two-phase annotation process: Sentence Phase for translation and Terminology Phase for tagging specific jargon.
  1. Sentence Phase: Korean posts were scraped from platforms like X (Twitter) and Theqoo. Expert human translators, vetted via a K-pop terminology test, translated these into English while preserving fandom-specific nuances.
  2. Terminology Phase: Experts categorized terms into three types: Group-Lexicon (unique vocabulary like poca), Group-NE (idol/group names), and Slang (internet variations).

The final benchmark consists of 1,000 parallel Korean-English sentence pairs containing 1,035 tagged terminologies.

Key Insights: Identity over Fluency

A critical finding of the study is that community connection often outweighs grammatical fluency. In a human survey, 89.75% of K-pop fans preferred translations that utilized fandom-specific terms over "correct" standard English. This suggests that for social groups, the identity signaled by jargon is a primary requirement for a successful translation.

However, models struggle to bridge this gap. Even when prompted to "act like a K-pop fan," GPT-4 achieved only a 16.3% accuracy rate on "Group-Lexicon" terms, indicating that these terms are used in ways current LLMs have not fully internalized.

Results and Evaluation

Models were evaluated using Exact-match term accuracy (EMA) and standard translation quality scores (BLEU/COMET):

  • GPT-4 led in terminology accuracy with an EMA of 26.4, though it still failed to capture nearly 75% of specialized terms.
  • Google Translate produced the most fluent English (BLEU 13.9) but struggled with nuance, yielding an EMA of only 10.4.
  • Back-translation using fandom-specific monolingual data surprisingly decreased performance in some cases, likely due to the "noisy" nature of fandom data confusing the models' comprehension.

Conclusion and Future Work

KpopMT highlights a significant limitation in current MT systems regarding social dialects. While this dataset treats K-pop as a broad category, future work aims to expand into group-specific terminologies (e.g., BTS-specific or Seventeen-specific slang) and apply this methodology to other social communities like sports or film fandoms to improve the balance between translation fluency and terminological accuracy.

Summary updated

Abstract

While machines learn from existing corpora, humans have the unique capability to establish and accept new language systems. This makes human form unique language systems within social groups. Aligning with this, we focus on a gap remaining in addressing translation challenges within social groups, where in-group members utilize unique terminologies. We propose KpopMT dataset, which aims to fill this gap by enabling precise terminology translation, choosing Kpop fandom as an initiative for social groups given its global popularity. Expert translators provide 1k English translations for Korean posts and comments, each annotated with specific terminology within social groups' language systems. We evaluate existing translation systems including GPT models on KpopMT to identify their failure cases. Results show overall low scores, underscoring the challenges of reflecting group-specific terminologies and styles in translation. We make KpopMT publicly available.
BibTeX
@misc{kim2024kpopmt,
  author = {JiWoo Kim and Yunsu Kim and JinYeong Bak},
  title = {{KpopMT: Translation Dataset with Terminology for Kpop Fandom}},
  year = {2024},
  eprint = {2407.07413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.07413},
}
Figure