TL;DR
This paper introduces KOTE, a large-scale Korean emotion dataset with 43 distinct emotions, addressing the limitations of existing Korean emotion corpora. KOTE was used to fine-tune a Korean language model, achieving improved performance over models trained on translated English datasets, demonstrating the value of culturally specific emotion taxonomies.
What Problem Does This Paper Address?
Sentiment analysis, while useful for classifying text as positive or negative, often falls short of capturing the nuanced spectrum of human emotions. Existing Korean emotion datasets are typically small and cover a limited range of emotions, hindering the development of sophisticated Korean emotion analysis tools. While English datasets like GoEmotions offer more detailed emotion taxonomies, directly applying them to Korean text via machine translation can lead to inaccuracies due to cultural differences in emotional expression. This paper aims to bridge this gap by creating a culturally relevant and comprehensive Korean emotion dataset.
Methods
The authors developed a novel emotion taxonomy for Korean by:
- Identifying Emotion Words: Compiling a large set of Korean words related to emotions from existing lexicons and manually adding emotional slang and emoticons.
- Word Vectorization: Using a pre-trained fastText model to create vector representations for these emotion words.
- Clustering: Employing a two-step clustering process (UMAP for dimension reduction and HDBSCAN for clustering) to identify underlying emotion structures. This process led to an initial identification of 30 clusters, which was then expanded to 136 potential clusters.
- Cluster Ensemble: Using a cluster ensemble method (HBGF) to aggregate the 136 clusters into a final taxonomy of 43 interpretable emotions.
The KOTE dataset itself was compiled by collecting 50,000 Korean online comments from 12 diverse platforms. These comments were then manually labeled for the 43 identified emotions through crowdsourcing, with multiple raters per comment to ensure label reliability.
Datasets
- KOTE (Korean Online That-gul Emotions) Dataset: A newly introduced dataset comprising 50,000 Korean online comments, each labeled with up to 43 emotions and "NO EMOTION." This dataset is the primary contribution of the paper.
- Existing Korean Emotion Datasets: The paper references several existing Korean emotion datasets (List of Korean Emotion Terms, Korean Emotion Vocabulary, KOSAC, NSMC, KNU SentiLex, Korean Continuous Dialogue Dataset with Emotion Information, Korean One-off Dialogue Dataset with Emotion Information, Emotional Dialogue Corpus) for comparison and to highlight their limitations (small size, coarse taxonomies).
- GoEmotions Dataset: An English emotion dataset used as a baseline for comparison. The paper notes that KOTE outperforms models trained on translated GoEmotions.
Key Insights and Ideas
- Cultural Specificity of Emotion: Emotions are deeply intertwined with culture, necessitating the development of culturally pertinent emotion taxonomies rather than relying on translations of existing ones.
- Complexity of Emotion: The study suggests that emotion is a complex structure, not easily captured by a small number of basic emotions, and that a taxonomy of 43 emotions is more effective than a smaller set.
- Leveraging Machine Learning for Taxonomy: Machine learning techniques, particularly clustering on word embeddings, can systematically derive nuanced emotion taxonomies from linguistic data.
- Bias in Language Data: The study highlights that even large datasets can reflect societal biases, such as gender bias, which can be detected and partially mitigated through methods like token switching.
Key Results
- KOTE Dataset: A comprehensive dataset of 50,000 Korean online comments with labels for 43 emotions.
- Fine-tuning Performance: A Korean language model (KCELECTRA) fine-tuned on KOTE achieved an F1-score of 0.56, significantly outperforming a model fine-tuned on translated GoEmotions (F1-score of 0.41).
- Gender Bias Detection: Analysis revealed that the source data exhibits gender bias, with texts containing female gender tokens being evaluated more negatively than those with male gender tokens, particularly for intense negative emotions.
- Bias Mitigation: Augmenting the training data with gender-swapped texts using token switching effectively reduced this gender bias without significantly impacting model performance.
Limitations and Future Work
- Emotion Complexity: The authors acknowledge that even 43 categories may not perfectly capture the full complexity and dynamic nature of emotions, which can interact and combine in intricate ways.
- Domain Specificity: While KOTE covers a broad range of online platforms, it may have limitations when applied to texts from very different domains or contexts (e.g., formal writing, specific professional jargon).
- Data Representation: The study notes that certain emotions, like "fear," might be underrepresented in the dataset, potentially limiting the model's ability to accurately detect them.
- Future Work: The authors suggest further research into more sophisticated methods for bias mitigation, exploring different preprocessing techniques (label merging, balancing), and applying the KOTE dataset to various downstream tasks. They also recommend users consider the ethical implications of using the dataset, particularly regarding potential biases reflected in the data.
Summary updated