Abstract
While machines learn from existing corpora, humans have the unique capability to establish and accept new language systems. This makes humans form unique language systems within social groups. Aligning with this, we focus on a gap remaining in addressing translation challenges within social groups, where in-group members utilize unique terminologies. We propose the KpopMT dataset, which aims to fill this gap by enabling precise terminology translation, choosing Kpop fandom as an initiative for social groups given its global popularity. Expert translators provide 1,000 English translations for Korean posts and comments, each annotated with specific terminology within social groups' language systems. We evaluate existing translation systems, including GPT models, on KpopMT to identify their failure cases. Results show overall low scores, underscoring the challenges of reflecting group-specific terminologies and styles in translation. We make KpopMT publicly available.
Introduction: The Terminology Gap in Social Dialects
Machine translation (MT) has reached high proficiency in standard languages but often falters when encountering "social dialects"—the specialized linguistic systems and jargon developed within specific online communities. Within the global Kpop fandom, terms like "stan," "bias," and "moot" carry specific social weight that general-purpose translators often fail to capture.
The authors identify a "terminology gap": in-group members use words that have specific meanings within their community which general-purpose translators often render literally or incorrectly. For example, translating a post about "stanning a bias" as "focusing on a favorite" misses the community-specific identity. KpopMT provides a benchmark for translating these culturally rich, niche language systems.
Methodology: Building KpopMT
The researchers developed a multi-phase pipeline to create a high-quality, terminology-tagged dataset:
- Data Collection: Korean posts and comments were scraped from fan community sites (Theqoo, Instiz) and Twitter (X) using a query list of fandom terms.
- Expert Translation: Ten translators fluent in Kpop jargon produced 1,000 English translation pairs. Translators were specifically instructed to use English fandom equivalents (like "stan" or "pc" for photo card) rather than standard English.
- Terminology Tagging: The dataset includes a "termbase" (glossary) where terms are categorized into three types: Group-Lexicon (unique fandom vocabulary), Group-NE (named entities like idol nicknames), and Slang (general internet variations).
Key Insights and Results
Evaluation of several models, including GPT-3.5, GPT-4, and Google Translate, revealed significant challenges in handling social dialects:
- GPT-4 as Terminology Leader: GPT-4 achieved the highest Exact-Match term Accuracy (EMA) of 26.4, significantly outperforming Google Translate (10.4). It excelled at common terms like "bias" and "stan."
- The Fluency vs. Jargon Trade-off: While Google Translate maintained the highest BLEU score (13.9), it often missed community-specific jargon. This indicates that models optimized for standard fluency may struggle with social authenticity.
- Social Connection Priority: A human survey of native English-speaking Kpop fans found that 89.75% preferred translations using fandom terms over standard language. Using the "right" jargon makes fans feel more connected to the community.
- Overall Difficulty: The fact that the top BLEU score was only 13.9 (compared to much higher scores on standard datasets) underscores just how difficult social dialects are for current AI.
Conclusion and Future Work
The KpopMT dataset highlights that "Kpop fandom" is not a monolith, as different groups have their own distinct sub-dialects. Future work aims to expand KpopMT to include other social groups, such as sports fans and global movie communities, to further refine how MT handles the diverse ways humans build identity through language.
Summary updated