How Intense Are You? Predicting Intensities of Emotions and Sentiments using Stacked Ensemble [Application Notes]

Md. Shad Akhtar, Asif Ekbal, Erik CambriaIEEE Comput. Intell. Mag 2020 · DOI · OpenReview

Introduction

Traditional sentiment analysis tools are often "coarse-grained," categorizing text into binary or ternary classes like happy, sad, or neutral. However, human emotion exists on a spectrum. There is a significant semantic and psychological gap between feeling "good" and "awesome," or "disappointed" and "devastated." This paper addresses the challenge of predicting the intensity of emotions and sentiments on a continuous scale, treating the task as a regression problem rather than a simple classification one.

The Challenge of Intensity Prediction

Predicting emotional nuances is inherently more difficult than classification due to several factors:

  1. Subjectivity: Intensity is highly personal and context-dependent.
  2. Noisy Data: Social media text is characterized by slang, abbreviations (e.g., "grt" for "great"), and "elongated" words (e.g., "goooood") that distort standard linguistic features.
  3. Domain Specificity: The polarity and intensity of words change across domains; for instance, "hot" carries different weight in a restaurant review compared to a technical report on hardware performance.

Methodology: The Stacked Ensemble Approach

To capture these nuances, the authors propose a Stacked Ensemble model. This architecture leverages the diverse strengths of multiple base models to produce a more accurate final prediction.

Base Models and Meta-Learner

The system utilizes four distinct base models:

  • Deep Learning Models: Three neural architectures—Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Gated Recurrent Units (GRU)—are trained on distributed word representations to capture local patterns and long-range dependencies.
  • Classical SVR: A Support Vector Regression (SVR) model utilizes hand-crafted features, including Tf-Idf, character n-grams, and specialized sentiment lexicons like VADER and SentiWordNet.
The proposed stacked ensemble architecture combines deep learning and classical regression models through a multi-layer perceptron meta-learner.

The outputs from these four models are fed into a three-layer Multi-Layer Perceptron (MLP). This meta-learner acts as a blender, learning which base model is most reliable for specific types of input text.

Denoising Auto-Encoder Word Embeddings (DAWE)

A key innovation in this work is the development of Denoising Auto-Encoder Word Embeddings (DAWE). To create a more robust representation, the authors concatenate GloVe and Word2Vec vectors. They then introduce "salt-and-pepper" noise (at a 60% rate) and use an auto-encoder to reconstruct the original signal. This process forces the model to learn the most salient semantic features, effectively filtering out the noise common in social media text.

The DAWE architecture concatenates multiple embedding types and uses a denoising auto-encoder to create a robust, cleaner representation.

Datasets and Evaluation

The model was rigorously evaluated on two distinct benchmark datasets to test its generalizability:

  1. EmoInt-2017 (WASSA-2017): A collection of generic tweets annotated for the intensity (0 to 1) of four primary emotions: anger, fear, joy, and sadness.
  2. SemEval-2017 Task 5: A dataset of financial short texts from microblogs (StockTwits/Twitter) and news headlines, with sentiment scores ranging from -1 (bearish) to +1 (bullish).

For the financial task, the authors further enhanced the model by crawling 126,000 news articles to generate domain-specific embeddings, as general-purpose embeddings often fail to capture the specific polarity of financial jargon.

Key Results and Insights

The proposed ensemble achieved state-of-the-art (SOTA) results across both domains:

  • Financial Sentiment: The model reached a cosine similarity of 0.797 on microblogs and 0.786 on news headlines, outperforming previous SOTA systems by 2.0 and 4.1 points, respectively.
  • Emotion Intensity: The system achieved an average Pearson correlation of 0.748 across the four target emotions.

The authors' analysis confirmed the Power of Complementarity: while CNNs might excel at identifying local emotional triggers, SVR models are more effective when specific lexicon-based "trigger words" are present. The MLP meta-learner successfully captures these complementary successes. Statistical t-tests (p < 0.05) confirmed that the ensemble's improvements over individual base models were significant.

Limitations and Future Directions

While highly effective, the model still faces challenges in specific linguistic scenarios:

  • Implicit Sentiment: Phrases like "I'm such a shy person" imply fear or anxiety without using explicit emotion words, leading to underestimation by the model.
  • Negation and Metaphor: Sarcastic or metaphorical language (e.g., "keep that fire burning" in a context labeled as anger) remains difficult for current architectures to interpret correctly.
  • Mixed Emotions: The current study focuses on single-emotion intensity. Future work will explore multi-emotion intensity and the integration of these high-fidelity sentiment scores into automated stock market prediction systems.

Summary updated

Abstract

Emotions and sentiments are subjective in nature. They differ on a case-to-case basis. However, predicting only the emotion and sentiment does not always convey complete information. The degree or level of emotions and sentiments often plays a crucial role in understanding the exact feeling within a single class (e.g., `good' versus `awesome'). In this paper, we propose a stacked ensemble method for predicting the degree of intensity for emotion and sentiment by combining the outputs obtained from several deep learning and classical feature-based models using a multi-layer perceptron network. We develop three deep learning models based on convolutional neural network, long short-term memory and gated recurrent unit and one classical supervised model based on support vector regression. We evaluate our proposed technique for two problems, i.e., emotion analysis in the generic domain and sentiment analysis in the financial domain. The proposed model shows impressive results for both the problems. Comparisons show that our proposed model achieves improved performance over the existing state-of-the-art systems.
BibTeX
@article{akhtar2020how,
  author = {Md. Shad Akhtar and Asif Ekbal and Erik Cambria},
  title = {{How Intense Are You? Predicting Intensities of Emotions and Sentiments using Stacked Ensemble [Application Notes]}},
  journal = {IEEE Comput. Intell. Mag.},
  year = {2020},
  doi = {10.1109/mci.2019.2954667},
}
Figure