EmoNLP at IEST 2018: An Ensemble of Deep Learning Models and Gradient Boosting Regression Tree for Implicit Emotion Prediction in Tweets

Man LiuEMNLP 2018

TL;DR

This paper presents an ensemble system for predicting implicit emotions in tweets, a challenging task where the emotion isn't explicitly stated. They combine three distinct models—a Convolutional Bidirectional LSTM (ConBLSTM), a Feed-Forward Neural Network (FFNN), and a Gradient Boosting Regression Tree (GBM)—to achieve robust performance. Their ensemble approach, particularly with optimized weights, significantly outperforms individual models and a baseline, demonstrating the power of combining diverse machine learning techniques for nuanced sentiment analysis.

What Problem Does This Paper Address?

Predicting emotions in text is a well-studied area, but this paper tackles a more subtle challenge: implicit emotion prediction. Unlike explicit emotion detection where words like "happy" or "sad" are present, implicit emotion prediction requires inferring feelings from context when the emotional word itself has been removed. This is crucial because real-world communication often relies on subtle cues rather than direct emotional declarations. The task, as defined by the IEST 2018 shared task, involves identifying one of six emotion types (anger, joy, fear, surprise, disgust, sad) when a placeholder like "[#TARGETWORD#]" indicates a missing emotional term. This is a significant step beyond simply keyword spotting and requires deeper semantic understanding.

Methods

The authors employ a multi-pronged strategy, building three distinct models and then ensembling them:

  • Convolutional Bidirectional LSTM (ConBLSTM): This model leverages deep learning to capture sentence-level and lexical features. It uses pre-trained word embeddings (trained on a massive dataset of 470 million tweets filtered for emotional words and emojis) and then feeds these embeddings through convolutional layers to extract local patterns, followed by a bidirectional LSTM to process sequential information. A softmax layer outputs emotion probabilities.
  • Feed-Forward Neural Network (FFNN): Inspired by prior work on emotional intensity prediction, this model uses a rich set of hand-crafted features alongside learned representations. It vectorizes word and character n-grams, and importantly, incorporates a wide array of sentiment features. These include POS tags, cluster features, negation features, word n-grams, counting features, and lexicon features derived from various sentiment lexicons (e.g., NRC Emotion Lexicon, MPQA Subjectivity Lexicon). These diverse features are concatenated to form a fixed-length sentence vector.
  • Gradient Boosting Regression Tree (GBM): This model utilizes the same concatenated feature set as the FFNN. GBM is chosen for its known efficiency and ability to handle broad sets of features. The authors tune hyperparameters like the number of trees and leaves per tree to optimize performance.

The core of their system is the ensemble of these three models. They combine the predicted probabilities from each model using a weighted average. While their submitted system used fixed weights, they later experimented with optimized weights (2 for FFNN, 2 for ConBLSTM, and 1 for GBM) to achieve better results.

Datasets

  • Primary Dataset: The IEST 2018 Implicit Emotion Shared Task dataset. This dataset includes 153,383 tweets for training, 9,591 for development, and 28,757 for testing. The task involves predicting one of six emotion types when an emotional word is removed.
  • External Dataset: A large corpus of 640 million tweets. This external data was used to train the word embeddings for the ConBLSTM model, providing a richer semantic representation than embeddings trained solely on the task-specific data.

Key Insights and Ideas

  • Ensembling for Robustness: The paper strongly advocates for ensembling diverse models. Combining a deep learning approach (ConBLSTM), a feature-rich neural network (FFNN), and a tree-based model (GBM) proved more effective than any single model, suggesting that each model captures different aspects of the implicit emotion signal.
  • Importance of Pre-trained Embeddings: For the ConBLSTM, training word embeddings on a massive, emotion-relevant corpus significantly boosted performance, highlighting the value of leveraging external, large-scale data for feature representation.
  • Feature Engineering Still Matters: The FFNN and GBM models demonstrate that carefully engineered features, especially those derived from various sentiment lexicons and linguistic properties, can still be highly effective, even in the age of deep learning.
  • Implicit Emotion is Hard: The results, while good, show that predicting implicit emotions is still a challenging task, with scores generally lower than those for explicit emotion detection.

Key Results

The ensemble model achieved the best performance, with an F1 score of 0.64 on the test set. This outperformed individual models like Feed Forward NN (0.62), ConBLSTM (0.55), and GBM (0.62), as well as a combined FFNN+BLSTM model (0.63). The ensemble also significantly beat the official WEKA Baseline (0.60).

Among individual emotions, the system performed best on "Joy" (F1 of 0.74 on test), which also happened to be the most frequent label in the dataset. This suggests that models tend to perform better on more prevalent classes.

Limitations and Future Work

  • Time Constraints: The authors mention that due to time limitations, they couldn't perform global optimization of ensemble weights during the submission phase. Post-submission tuning with optimized weights led to better results.
  • Feature Analysis: They acknowledge that a detailed analysis of which specific features contributed most to the FFNN and GBM models was not conducted, which could lead to further improvements.
  • Future Directions: The authors plan to experiment with different word embedding dimensions and methods, explore more ensemble techniques, and investigate the effectiveness of individual feature sets more thoroughly. They also aim to test more ensemble methods and analyze the contribution of each approach within the ensemble.

Summary updated

BibTeX
@inproceedings{liu2018emonlp,
  author = {Man Liu},
  title = {{EmoNLP at IEST 2018: An Ensemble of Deep Learning Models and Gradient Boosting Regression Tree for Implicit Emotion Prediction in Tweets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure