UTFPR at IEST 2018: Exploring Character-to-Word Composition for Emotion Analysis

Gustavo PaetzoldEMNLP 2018

TL;DR

This paper introduces the UTFPR system, a character-to-word recurrent neural network for emotion analysis in tweets. It tackles the challenge of informal language and orthographic variance in social media without relying on external resources, offering an elegant and effective solution.

What Problem Does This Paper Address?

Emotion analysis, a crucial task in Natural Language Processing (NLP), aims to understand the emotional tone of text. While effective for formal writing, analyzing informal text like tweets presents significant challenges. Tweets often feature misspellings, non-standard abbreviations, and the use of emojis, making it difficult for traditional NLP models to accurately capture sentiment. Existing state-of-the-art systems for tweet-based emotion analysis, like SeerNet, often rely on extensive external resources such as specialized tokenizers, embedding models, and emoji analyzers, making them complex and hard to replicate, especially for under-resourced languages. This paper addresses the need for a simpler, yet effective, system that can handle the inherent variability of tweet language without extensive external dependencies.

Methods

The core of the UTFPR system is a compositional character-to-word recurrent neural network (RNN). The approach works as follows:

  1. Character Embeddings: Words to the left and right of a target emotion word (marked as [#TRIGGERWORD#]) are first processed by a character embedding layer. This layer converts individual characters into dense vector representations.
  2. Character-to-Word RNN: These character embeddings are then fed into a bidirectional GRU (Gated Recurrent Unit) network. This layer learns to represent each word by composing its character-level information.
  3. Word-to-Sentence RNNs: The word representations from the character-to-word layer are then passed to two separate bidirectional GRU networks, one for the left context and one for the right context. These networks generate a numerical representation for each context.
  4. Concatenation and Prediction: The final representations of the left and right contexts are concatenated and fed into a perceptron layer, which outputs the predicted emotion label using a softmax function.

This architecture is designed to be robust to orthographic variance because it learns word representations from their constituent characters, allowing it to generalize across different spellings of the same word. The system is trained end-to-end and does not require external lexicons or pre-trained models beyond the input text itself.

Datasets

The primary dataset used is from the Implicit Emotions Shared Task (IEST) of 2018, hosted as part of the WASSA 2018 workshop. This dataset consists of:

  • A training set of 153,383 instances.
  • A trial set of 9,591 instances.
  • An unlabeled test set of 28,757 instances.

Each instance includes a tweet with a target emotion word replaced by [#TRIGGERWORD#] and an associated emotion label. The six possible emotions are joy, sad, disgust, anger, surprise, and fear.

For preliminary experiments, the authors also generated a synthetic dataset of 1,774,423 automatically extracted complementary instances by finding morphological variants and synonyms of the target emotion words in a large corpus and replacing them with the [#TRIGGERWORD#] marker.

Key Insights and Ideas

  • Character-level composition is key for handling orthographic variance: The paper strongly demonstrates that learning word representations from characters is highly effective in dealing with the noisy and varied spelling found in tweets.
  • Simplicity can be powerful: The UTFPR system achieves competitive results without relying on complex external resources, making it more accessible and easier to adapt to new domains or languages.
  • Contextual modeling is crucial: Separately modeling the left and right contexts of the target word, as opposed to treating the entire tweet as a bag-of-words, proved to be more effective in preliminary experiments.
  • Dropout enhances robustness: The inclusion of dropout in the model significantly improves its performance, particularly in handling noisy or "jammed" versions of the test data.

Key Results

  • The final UTFPR system achieved a macro F-score of 0.569 on the IEST 2018 test set, placing it 23rd in the competition. While not a top-tier result, it demonstrates the effectiveness of their approach.
  • Preliminary experiments showed that modeling words to the left and right of the target word separately, without data enhancement, yielded the most promising results.
  • A "jammed" test set experiment revealed that the complete UTFPR system (with character-to-word layers and dropout) was significantly more robust to orthographic variations than its variants (UTFPR-C and UTFPR-CD) that lacked these features. The performance drop for the full UTFPR system was only 2.2% compared to over 14.3% for the other variants.

Limitations and Future Work

  • Limited Computing Resources: The authors acknowledge that due to limited computing resources, they could not train more variants of their system or explore additional design decisions.
  • Performance Gap: The UTFPR system, while elegant and robust, did not reach the performance levels of more elaborate, resource-heavy state-of-the-art approaches.
  • Future Work: The authors aim to create more sophisticated variants of the UTFPR approach by incorporating other cost-effective sources of information to further improve performance. They also suggest exploring more advanced recurrent architectures or attention mechanisms.

Summary updated

BibTeX
@inproceedings{paetzold2018utfpr,
  author = {Gustavo Paetzold},
  title = {{UTFPR at IEST 2018: Exploring Character-to-Word Composition for Emotion Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure