Amobee at IEST 2018: Transfer Learning from Language Models

Alon Rozental, Daniel Fleischer, Zohar KelrichEMNLP 2018

TL;DR

This paper presents Amobee's winning system for the IEST 2018 implicit emotion shared task, which achieved first place by leveraging pre-trained language models to predict emotions in tweets that don't explicitly state them. Their novel approach uses language models to estimate the probability of different emotions, combined with an ensemble of deep learning models, to outperform other systems.

What Problem Does This Paper Address?

The paper tackles the challenge of implicit emotion detection in social media, specifically on Twitter. Unlike traditional sentiment analysis that looks for explicit emotion words (like "happy" or "sad"), this task requires identifying emotions that are implied or suggested by the context of a tweet, even when no direct emotion-related words are present. This is a harder problem because it requires a deeper understanding of language nuances, sarcasm, and subtle expressions. The IEST 2018 shared task aimed to classify tweets into one of six emotions: anger, disgust, fear, joy, sadness, and surprise, without relying on explicit emotion keywords.

Methods

The core of Amobee's approach is the innovative use of pre-trained language models (LMs) as emotion classifiers. They trained a Transformer-Decoder based language model (LM1) on a massive dataset of 5 billion tweets (the "Tweets 5B dataset"). This LM was then used to predict the probability of a tweet being generated, and by substituting potential emotion words into a tweet, they could infer the likelihood of different emotions.

Key methodological components include:

  • Language Model Training: A Transformer-Decoder architecture was trained on a large Twitter corpus. This model was then fine-tuned on the shared task data to create a second LM (LM2).
  • Emotion Prediction with LMs: For each tweet, they generated a list of possible emotion words (from a lexicon) and replaced the placeholder trigger word with each of these. The LM then provided probabilities for each emotion, which were normalized to represent the LM's confidence in each emotion.
  • Ensemble System: The system is built upon a multi-level ensemble.
  • Mini ASC Modules: These are bi-LSTM layers with CNN-based attention mechanisms, similar to previous work by Amobee.
  • First Ensemble (E1): This level combines predictions from multiple Mini ASC modules with outputs from other networks, including those trained on Semeval 2018 datasets and the LM predictions.
  • Second Ensemble (E2): This level further combines the outputs of the first ensemble with external embeddings (Universal Sentence Embedding, DeepMoji) and Semeval 2018 pipeline predictions.
  • Feature Engineering: Beyond LM predictions, they incorporated features from external embeddings and predictions from their previous Semeval 2018 system. They also created binary features to identify trigger words.

Datasets

The paper utilized several datasets:

  • Tweets 5B Dataset: A random sample of 5 billion tweets from the Twitter Firehose, used for training language models and word embeddings.
  • SemEval 2018 Task 1 and Task 5 Datasets: Used for training and generating features. These datasets involved classifying tweets into emotions.
  • Official IEST 2018 Task Datasets: The primary datasets for training and evaluation of the implicit emotion task. Table 1 provides a distribution of labels across train, dev, and test sets for these datasets.

Key Insights and Ideas

  • Language Models as Emotion Predictors: The most significant insight is the successful application of pre-trained language models, not just for text generation or understanding, but as direct classifiers for implicit emotions. This demonstrates their capability to capture subtle semantic cues.
  • Ensembling for Robustness: The multi-level ensemble approach, combining diverse models (LMs, LSTMs, CNNs) and features, proved crucial for achieving high performance and robustness.
  • Data Splitting Strategy: The authors found that splitting the dataset into training and validation sets for different stages of the ensemble was more beneficial than training on the entire combined dataset, especially for models that might overfit.
  • Human Performance Benchmark: The paper notes that the task organizers tested human performance, achieving a macro F1 of 0.45, which is significantly lower than the automated systems, highlighting the difficulty of the task.

Key Results

Amobee's system achieved 1st place in the IEST 2018 implicit emotion shared task, with a macro F1 score of 0.7145 on the test dataset. This was a narrow victory, with the second-place system scoring 0.7105. The paper also reports validation F1 scores for various components, such as 0.700 for the first ensemble (E1) and 0.705 for the final soft voting ensemble before final training.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations of the LM approach itself, but the ensemble nature suggests that individual models might have weaknesses. The use of a fixed lexicon for emotion words could also be a limitation if it doesn't cover all nuances.
  • Future Work: The authors plan to release the word embeddings and language models as open-source to facilitate further research and resource sharing in the NLP community. This suggests a commitment to advancing the field beyond their specific task.

Summary updated

BibTeX
@inproceedings{rozental2018amobee,
  author = {Alon Rozental and Daniel Fleischer and Zohar Kelrich},
  title = {{Amobee at IEST 2018: Transfer Learning from Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure