Sentiment classification in English from sentence-level annotations of emotions regarding models of affect

Alexandre Trilla, Francesc AlíasInterspeech 2009

TL;DR

This paper presents a novel approach to automatically classifying the sentiment (positive/negative) of text by leveraging sentence-level emotion annotations and mapping them to established affective models. By demonstrating that a fine-grained emotion model (Russell's circumplex) leads to better sentiment classification than simple valence scores, the work paves the way for more natural expressive text-to-speech (TTS) systems.

What Problem Does This Paper Address?

Understanding and recognizing human emotions in text is crucial for creating more natural and engaging human-computer interactions, particularly in applications like expressive Text-to-Speech (TTS) synthesis. While previous work has focused on identifying emotions in text, this paper tackles the challenge of accurately classifying the sentiment (positive vs. negative) of text by exploring how different models of affect can inform this classification. The authors highlight the limitations of existing systems, such as EmoLib, which rely on simpler rule-based approaches and may struggle with nuanced emotional expression. They aim to bridge the gap between basic emotion recognition and sentiment analysis, ultimately contributing to more sophisticated affective computing.

Methods

The core of this paper lies in its novel approach to sentiment classification by mapping sentence-level emotion annotations to established affective models.

  • EmoLib Enhancement: The authors build upon the existing EmoLib system, which uses a pipeline of Natural Language Processing (NLP) modules (lexical analyzer, POS tagger, word-sense disambiguator, stemmer, keyword spotter) and a rule-based classifier to identify emotions.
  • Affective Model Mapping: The key innovation is mapping the Semeval 2007 dataset, which contains sentence-level emotion annotations, into different models of affect. They specifically evaluate three models:
  • Russell's Circumplex Model: A bidimensional space representing valence (positive/negative) and activation (stimulation).
  • Whissell's Dictionary of Affect: A model that arranges emotions in an "emotion wheel."
    • Scherer's Theory of Affect: Another model used for binary classification experiments.
  • Sentiment Classification: After identifying the most suitable affective model (Russell's), they group emotions into "positive" (neutral, happiness, surprise) and "negative" (anger, fear, sorrow) classes. A hard binary classifier is then trained using the centroids of these two sentiment classes.
  • Evaluation: A 10-fold cross-validation procedure is employed to evaluate the effectiveness of the proposed sentiment classification approach.

Datasets

  • Semeval 2007 Dataset: This dataset, consisting of 250 headlines (later expanded to 1250 for training and testing), was used for evaluation. The headlines were annotated with six different emotions by multiple evaluators. This dataset is crucial as it provides sentence-level emotion annotations, which is central to the paper's methodology.

Key Insights and Ideas

  • Fine-grained Emotion Models Improve Sentiment Classification: The most significant finding is that using a more nuanced model of affect, like Russell's circumplex, leads to better sentiment classification accuracy compared to relying solely on positive/negative valence scores. This suggests that understanding the underlying emotional dimensions provides richer information for sentiment analysis.
  • Russell's Model is Well-Suited for the Semeval Dataset: Among the evaluated affective models, Russell's circumplex model proved to be the most effective for representing the Semeval 2007 dataset, resulting in balanced distributions and successful classification rates.
  • Sentence-Level Annotations are Key: The paper emphasizes the importance of sentence-level emotion annotations for building robust sentiment classifiers, moving beyond word-level analysis.
  • Limitations of Simple Valence: The study confirms the intuition that while valence is important, it's not sufficient. A direct comparison showed that using fine-grained emotion annotations mapped to a circumplex model yielded a higher accuracy (around 61%) than using only valence scores (around 57%).

Key Results

  • Sentiment Classification Accuracy: The proposed sentiment classifier, based on mapping to Russell's model and using a hard binary classifier with centroids, achieved an approximate accuracy of 61% (F1 score of 61.25% ± 6.12).
  • Comparison with Valence-Only Approach: When using only valence scores for sentiment classification, the accuracy dropped to approximately 57% (F1 score of 57.34% ± 5.95), confirming the benefit of the fine-grained emotion approach.
  • EmoLib's Original Performance: The original EmoLib rule-based classifier, when adapted, showed limitations, with some emotions (surprise and sorrow) having 0% prediction rates in its confusion matrix.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the Semeval 2007 dataset, while useful, is relatively small, which might limit the generalizability of the conclusions.
  • Neutral Bias: The system's default behavior of assigning a "neutral" label when no keywords are found can introduce bias. The paper notes that this approach, while common, can sometimes worsen performance.
  • Future Work:
    • More Extensive Emotional Lexicon: To reduce the reliance on the default neutral label, the authors suggest expanding the knowledge of emotional words.
  • Hierarchical Emotion Classification: They plan to extend the system to a hierarchical emotion classifier, moving beyond simple positive/negative sentiment to capture more nuanced emotional states.
  • Beyond Lexical Features: The authors recognize that emotions in text are not solely conveyed through lexical features. Future work will explore other linguistic structures to infer finer emotional predictions.
  • Integration with Expressive TTS: The ultimate goal is to apply this sentiment classification scheme to generate expressive synthetic speech, building upon previous work in multidomain TTS synthesis.

Summary updated

BibTeX
@inproceedings{trilla2009sentiment,
  author = {Alexandre Trilla and Francesc Alías},
  title = {{Sentiment classification in English from sentence-level annotations of emotions regarding models of affect}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure