UBC-NLP at IEST 2018: Learning Implicit Emotion With an Ensemble of Language Models

Hassan Alhuzali, Mohamed Elaraby, Muhammad Abdul-MageedEMNLP 2018

TL;DR

This paper presents UBC-NLP's approach to the IEST 2018 Implicit Emotion task, achieving a strong 4th place finish by leveraging pre-trained language models (LMs) and an ensemble of fine-tuned models. Their work highlights the significant impact of pre-trained LMs like ELMo and ULMFIT for capturing nuanced emotions in short, noisy text like tweets, even with limited training data.

What Problem Does This Paper Address?

Detecting emotion in text is crucial for understanding human communication and has applications in various domains. However, explicitly stated emotions are often absent in informal text like tweets, leading to the challenge of "implicit emotion detection." This task requires models to infer emotions from context, even when emotion-laden words are removed or implied. Existing methods often rely on manually curated datasets or simple keyword/hashtag matching, which can be brittle. The IEST 2018 "Implicit Emotion" shared task specifically aimed to address this by providing a dataset where trigger words for emotions were removed, forcing models to learn from the surrounding context.

Methods

The UBC-NLP team explored several approaches, categorizing them into three main strategies:

  • Tuning simple pre-trained embeddings: They used fastText embeddings, which capture sub-word information and are robust to out-of-vocabulary words. These were fed into LSTM and BiLSTM networks.
  • Tuning embeddings from language models: They utilized ELMo embeddings, which are deep contextualized word representations that capture syntax and semantics across different linguistic contexts. These were also used with LSTM and BiLSTM architectures.
  • Directly tuning language models: This was their most successful approach, employing the ULMFIT framework. ULMFIT involves pre-training a language model on a large corpus (like Wikipedia) and then fine-tuning it on the target task. They experimented with forward (Fwd), backward (Bwd), and an ensemble (BiLM) of these fine-tuned LMs. Key techniques within ULMFIT include "gradual unfreezing" and "discriminative fine-tuning" to prevent overfitting and improve performance.

Datasets

The primary dataset used was the Twitter dataset released for the IEST 2018 "Implicit Emotion" task. This dataset was partitioned into:

  • Training: 153,383 tweets
  • Validation: 9,591 tweets
  • Testing: 28,757 tweets

The task involved predicting one of six basic emotions (anger, disgust, fear, joy, sad, surprise) from tweets where the explicit emotion trigger word was removed.

Key Insights and Ideas

  • Pre-trained LMs are powerful for implicit emotion: The study strongly demonstrates that leveraging large, pre-trained language models like ELMo and ULMFIT significantly outperforms simpler embedding methods for implicit emotion detection. This is attributed to their ability to capture deeper contextual and semantic nuances.
  • ULMFIT excels in implicit emotion tasks: The ULMFIT framework, with its sophisticated fine-tuning strategies, yielded the best results, highlighting its effectiveness for tasks requiring contextual understanding beyond simple word matching.
  • Ensembling improves performance: Combining predictions from multiple fine-tuned LMs (BiLM) provided a performance boost, suggesting that different LM configurations capture complementary information.
  • Limited data can still yield good results: Surprisingly, their best model achieved strong performance even when trained on less than 40% of the available training data, underscoring the power of pre-training.
  • Emotion confusion is inherent: The error analysis revealed that certain emotion classes are frequently confused (e.g., anger with surprise, sadness with anger), likely due to overlapping lexical expressions and the inherent ambiguity of implicit emotions.

Key Results

  • The UBC-NLP system ranked 4th out of 30 teams in the IEST 2018 competition with an F-score of 69.3%.
  • Post-competition experiments achieved an even higher F-score of 70.7% with their BiLM (ensemble) model, which would have placed them in the top 3.
  • Their ULMFIT-based models consistently outperformed baselines using fastText and ELMo embeddings. For instance, the Fwd ULMFIT model achieved 69.4% F-score.
  • Even simple LSTM/BiLSTM models with ELMo embeddings achieved ~1.5% higher F-score than the logistic regression baseline (60%).
  • The model trained on only 10% of the data already outperformed the baseline, and using 40% of the data surpassed fastText and ELMo models.

Limitations and Future Work

  • Emotion Class Confusion: The paper acknowledges that the classifier struggles with distinguishing between certain emotion classes, which is attributed to the inherent difficulty of the task and potential overlap in lexical sequences between training and test data.
  • Deeper Architectures for Embeddings: The authors suggest that while ELMo embeddings performed well, deeper neural architectures could potentially further improve performance when using these embeddings.
  • Twitter-Specific Pre-training: Negative experiments showed that training ULMFIT on Twitter-specific data did not yield gains compared to Wikipedia pre-training, possibly due to the smaller size of the Twitter corpus. This suggests that the scale and diversity of the initial pre-training corpus are critical.
  • Future Work: The authors mention exploring deeper architectures and further investigating the impact of pre-training data scale and domain.

Summary updated

BibTeX
@inproceedings{alhuzali2018ubc,
  author = {Hassan Alhuzali and Mohamed Elaraby and Muhammad Abdul-Mageed},
  title = {{UBC-NLP at IEST 2018: Learning Implicit Emotion With an Ensemble of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure