EmoTag1200: Understanding the Association between Emojis and Emotions

Abu Awal Md Shoeb, Gerard de MeloEMNLP 2020 · OpenReview

Abstract: Abu Awal Md Shoeb, Gerard de Melo. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

While emojis are ubiquitous in digital communication, computational understanding of them has largely been limited to simple sentiment polarity (positive vs. negative). However, emojis often convey specific affective states that are difficult to express in text alone. The challenge lies in the lack of a comprehensive, human-validated dataset measuring the intensity of association between a wide variety of emojis and a broad spectrum of emotions. Furthermore, it remained unclear if automated, unsupervised methods could replicate human intuition regarding these associations without massive amounts of labeled training data.

The EmoTag1200 Dataset and Methodology

To address this, the researchers introduced EmoTag1200, a dataset consisting of 1,200 pairs (150 emojis × 8 emotions) with human-assigned association scores. They selected the 150 most frequent emojis from Twitter and asked nine human raters to score each against the eight basic emotions from Plutchik’s Wheel of Emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, and trust) on a 5-point scale.

In tandem, they developed an Emoji-Centric Tweet Corpus containing 20.8 million English tweets to train domain-specific embeddings. They tested several unsupervised prediction models, including distributed word vectors (GloVe and custom word2vec), lexicon-based mapping using the NRC Emotion Intensity Lexicon (NRC-EIL), and co-occurrence frequencies to determine how well machines could mimic human emotional attribution.

Key Results and Human Agreement

The study established a high bar for automated systems, finding an average Pearson correlation of 0.69 between human raters. Agreement was highest for "obvious" emotions like joy, sadness, and fear, while more subjective emotions like "trust" and "anticipation" saw significantly more disagreement among raters.

Average Pearson correlation between human raters across eight emotions.

Regarding automated performance, the best unsupervised model (NRC-EIL using co-occurrence frequencies) achieved an average correlation of 0.53. Notably, for the emotion "Anger," the model reached a correlation of 0.74, actually exceeding the average human inter-rater agreement (0.67) for that specific category. The study also highlighted that standard pre-trained GloVe models were ineffective (0.03 correlation), whereas custom word2vec models trained on emoji-centric data reached 0.45.

Insights into Emoji-Emotion Associations

A primary insight from the data is the "Joy Dominance": humans associate a vast and diverse range of objects and concepts with joy. In contrast, emotions like disgust or anger have a much more concentrated set of associated emojis, primarily limited to facial expressions.

Examples of emojis associated with specific emotions in the EmoTag1200 dataset.

The research also demonstrated that intensity is key. Models utilizing intensity scores (how strongly a word relates to an emotion) significantly outperformed those using binary labels (whether a word relates to an emotion at all).

Conclusion and Future Work

EmoTag1200 provides a vital resource for mapping the nuanced emotional landscape of emojis. While the current dataset focuses on 150 popular emojis among English speakers aged 25–35, it lays the groundwork for future scaling to the entire Unicode emoji set and exploring cross-cultural variations. These findings have significant implications for improving sentiment analysis, sarcasm detection, and context-sensitive emoji recommendation systems.

Summary updated

BibTeX
@inproceedings{shoeb2020emotag,
  author = {Abu Awal Md Shoeb and Gerard de Melo},
  title = {{EmoTag1200: Understanding the Association between Emojis and Emotions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=d_wiwUbPEZ},
}
Figure