An Analysis of Annotated Corpora for Emotion Classification in Text

Laura-Ana-Maria Bostan, Roman KlingerCOLING 2018

TL;DR

This paper tackles the challenge of inconsistent emotion classification datasets by creating a unified, standardized corpus from existing resources. They then conduct the first systematic cross-corpus experiments, revealing that models trained on one dataset don't always generalize well to others, and that a curated subset of data often outperforms using all available data.

What Problem Does This Paper Address?

Emotion classification in text is a growing area with applications in everything from chatbots to mental health analysis. However, the field is hampered by a proliferation of datasets that differ significantly in their annotation schemes (e.g., discrete emotions vs. continuous valence/arousal), the domains they cover (news, blogs, social media), and their file formats. This fragmentation means researchers often use only a limited subset of available resources, and there's a lack of systematic comparison between datasets. This makes it difficult to choose the best resources for developing a new emotion classification model, especially for a novel domain.

Methods

The authors' primary methodological contribution is the creation of a unified, aggregated corpus of emotion-annotated text data. They achieved this by:

  • Standardizing Annotation Schemas: They mapped various existing emotion labels to a common set of 11 labels (anger, anticipation, confusion, disgust, fear, joy, love, noemo, sadness, surprise, trust), largely following established psychological models like Plutchik's and Parrott's.
  • Common File Format: All datasets were converted into a consistent format, making them easier to access and process.
  • Cross-Corpus Experiments: They conducted experiments where models trained on one dataset were tested on others. This included both pairwise comparisons and an "all-vs-one" setting where a model was trained on all datasets except the test set.
  • Machine Learning Model: They employed Maximum Entropy (MaxEnt) classifiers with Bag-of-Words (BOW) features, using L2 regularization and instance weights for class balancing. This choice was motivated by previous work showing linear classifiers perform comparably to neural methods for this task, offering simplicity and reproducibility.

Datasets

The paper analyzes and aggregates 14 existing emotion-annotated datasets. These datasets vary widely in source (news headlines, blogs, tweets, conversations, descriptions), annotation schema, and size. Notable datasets include:

  • AffectiveText: News headlines, Ekman's emotions + valence.
  • Blogs: Blog sentences, Ekman's emotions.
  • CrowdFlower: Tweets, non-standard labels.
  • DailyDialogs: Conversations, Ekman's emotions + "no emotion."
  • Electoral-Tweets: Tweets, non-standard labels mapped to Plutchik.
  • EmoBank: Multi-genre sentences, valence-arousal-dominance model.
  • ISEAR: Descriptions of emotional events, Ekman's emotions + shame/guilt.
  • TEC (Twitter Emotion Corpus): Tweets, Ekman's emotions via hashtags.

The authors provide a script to download and convert these datasets into their unified format, addressing licensing restrictions where necessary.

Key Insights and Ideas

  • Dataset Heterogeneity is a Major Hurdle: The significant differences in annotation schemes and domains across datasets make direct model transfer challenging.
  • "Easier" Datasets Exist: Some datasets, like Emotion-Stimulus and EmoInt, are easier to classify within themselves, often because they were designed for related but distinct tasks (e.g., emotion intensity prediction).
  • Twitter Data is Noisy: Models generally perform worse on Twitter data compared to other domains, suggesting inherent noise or complexity in social media annotations.
  • Curated Subsets Outperform Aggregation: Surprisingly, using a subset of datasets for training often leads to better performance on a target dataset than training on the entire aggregated corpus. This suggests that not all data is equally beneficial, and some datasets might even introduce noise or irrelevant patterns.
  • Domain and Annotation Style Matter: The paper highlights how domain (e.g., news vs. tweets) and annotation procedure (expert vs. crowdsourced) influence model performance.

Key Results

  • Within-Corpus Performance Varies: Scores for within-corpus classification (training and testing on the same dataset) ranged significantly, with Emotion-Stimulus and EmoInt showing high scores, while CrowdFlower and Electoral-Tweets were more challenging.
  • Cross-Corpus Performance is Lower: Generally, F1-scores for cross-corpus experiments were lower than within-corpus results, indicating generalization challenges.
  • Blogs and DailyDialogs are Informative: Models trained on Blogs and DailyDialogs tended to perform well when tested on other datasets.
  • Electoral-Tweets is an Exception: This dataset showed similar performance whether trained on itself or on Blogs, suggesting a strong relationship or transferability.
  • "All-vs-One" is Not Always Best: The "all-vs-one" experiment (training on all datasets except the target) showed that adding more data doesn't always improve performance, especially for highly transformed datasets or those with complex label structures.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that the similarity measure used (cosine similarity of term distributions) is an approximation and has a moderate correlation with model performance.
  • For datasets with non-aligned labels, they used the intersection of labels, potentially discarding some information.
  • The study primarily uses MaxEnt classifiers, and the performance of more advanced neural models might differ.
  • Future Work:
    • The authors plan to release another version of their unified dataset that includes mappings between different emotion models and dimensional representations.
  • They aim to conduct transfer learning experiments across different domains and annotation procedures.
  • Further qualitative analysis of emotion realization across different annotation schemata and domains is proposed.

Summary updated

BibTeX
@inproceedings{bostan2018analysis,
  author = {Laura-Ana-Maria Bostan and Roman Klinger},
  title = {{An Analysis of Annotated Corpora for Emotion Classification in Text}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure