Abstract
KpopMT is a specialized translation dataset designed to bridge the gap between standard machine translation and the nuanced "social dialects" used by global fandoms. By providing 1,000 Korean-English sentence pairs annotated with K-pop-specific jargon, this research demonstrates that even state-of-the-art models like GPT-4 struggle to accurately translate the unique terminology that defines online communities.
The Challenge of Social Dialects
Machine translation (MT) has achieved high performance in general and technical domains, yet it remains remarkably poor at handling social dialects—the unique linguistic systems, slang, and terminologies that emerge within specific social groups.
In the K-pop fandom, specialized language (e.g., "stan," "bias," or "poca") is essential for community building. Standard MT systems often "correct" these terms into standard language or mistranslate them entirely because they are underrepresented in general training corpora. For instance, a fan's reference to their "bias" might be translated as "favorites," stripping the term of its specific cultural weight.
Methodology: The KpopMT Dataset
The researchers developed the KpopMT dataset using a rigorous two-phase approach to ensure both translation quality and terminological accuracy.
- Sentence Phase: Korean posts were scraped from platforms like X (Twitter) and Theqoo. Expert human translators, vetted via a K-pop terminology test, translated these into English while preserving fandom-specific nuances.
- Terminology Phase: Experts categorized terms into three types: Group-Lexicon (unique vocabulary like poca), Group-NE (idol/group names), and Slang (internet variations).
The final benchmark consists of 1,000 parallel Korean-English sentence pairs containing 1,035 tagged terminologies.
Key Insights: Identity over Fluency
A critical finding of the study is that community connection often outweighs grammatical fluency. In a human survey, 89.75% of K-pop fans preferred translations that utilized fandom-specific terms over "correct" standard English. This suggests that for social groups, the identity signaled by jargon is a primary requirement for a successful translation.
However, models struggle to bridge this gap. Even when prompted to "act like a K-pop fan," GPT-4 achieved only a 16.3% accuracy rate on "Group-Lexicon" terms, indicating that these terms are used in ways current LLMs have not fully internalized.
Results and Evaluation
Models were evaluated using Exact-match term accuracy (EMA) and standard translation quality scores (BLEU/COMET):
- GPT-4 led in terminology accuracy with an EMA of 26.4, though it still failed to capture nearly 75% of specialized terms.
- Google Translate produced the most fluent English (BLEU 13.9) but struggled with nuance, yielding an EMA of only 10.4.
- Back-translation using fandom-specific monolingual data surprisingly decreased performance in some cases, likely due to the "noisy" nature of fandom data confusing the models' comprehension.
Conclusion and Future Work
KpopMT highlights a significant limitation in current MT systems regarding social dialects. While this dataset treats K-pop as a broad category, future work aims to expand into group-specific terminologies (e.g., BTS-specific or Seventeen-specific slang) and apply this methodology to other social communities like sports or film fandoms to improve the balance between translation fluency and terminological accuracy.
Summary updated