SEntiMoji: an emoji-powered learning approach for sentiment analysis in software engineering

Zhenpeng Chen, Yanbin Cao, Xuan Lu, Qiaozhu Mei, Xuanzhe Liu

TL;DR

SEntiMoji addresses the chronic lack of labeled data in software engineering (SE) sentiment analysis by leveraging billions of emojis as "noisy labels." By fine-tuning a deep learning model on both general Tweets and domain-specific GitHub posts, the researchers created a tool that understands technical jargon and significantly outperforms existing state-of-the-art SE sentiment tools.

What Problem Does This Paper Address?

Sentiment analysis—identifying the emotional tone of text—is incredibly useful in software engineering. It helps project managers detect developer burnout in commit messages, understand collective opinions on Q&A forums like Stack Overflow, and gauge the usability of new APIs.

However, standard sentiment tools often fail when applied to SE data. The primary culprit is technical jargon. In a general context, the word "painful" is almost always negative. In a technical discussion like "Without tests, changes in Rails 3 may be very painful," the word describes a factual difficulty rather than a personal negative emotion.

To fix this, researchers usually build custom tools using manually labeled SE datasets. But these datasets are tiny (a few thousand samples) compared to the millions of examples needed to train robust deep learning models. Manual labeling is also slow, expensive, and prone to human error.

Methods

The authors propose SEntiMoji, an approach that uses emojis as a bridge between general sentiment and technical language. The core idea is that emojis are "noisy labels": if a developer uses a "thumbs up" emoji, the text is likely positive.

The architecture is based on DeepMoji, a model consisting of a word embedding layer, two bi-directional LSTM layers to capture context, and an attention mechanism to weigh important words.

The architecture of DeepMoji

The researchers used a two-stage learning process:

  1. Representation Learning: They started with a model pre-trained on 1.2 billion Tweets to predict emojis. They then fine-tuned this on ~1 million GitHub posts containing emojis. This taught the model how technical words (like "bug," "fix," or "refactor") relate to emotional expressions.
  2. Classifier Training: They replaced the final emoji-prediction layer with a sentiment polarity layer (Positive, Neutral, Negative). They then used a small amount of manually labeled SE data to fine-tune the entire network using a "chain-thaw" approach—unfreezing and training layers one by one to prevent the model from forgetting what it learned in the first stage.

Datasets

The study evaluated SEntiMoji against four representative benchmark datasets:

  • JIRA: Issue comments from the JIRA tracking system.
  • Stack Overflow: Questions, answers, and comments from the popular Q&A site.
  • Code Review: Comments extracted from code review repositories.
  • Java Library: Sentences specifically about Java APIs and libraries.
Benchmark datasets used in this study

Key Insights and Ideas

  • The "Twitter Surprise": One of the most striking findings was that general sentiment patterns from Tweets contributed more to the model's power than the domain-specific GitHub data. A version of the model trained only on Tweets and then fine-tuned on labeled SE data (SEntiMoji-T) performed nearly as well as the full version. This suggests that core emotional expressions are largely shared across domains.
  • Efficiency with Small Data: SEntiMoji is remarkably data-efficient. Even when trained on only 20% of the available labeled data, it still outperformed several existing tools that used 100% of the data.
  • Precision Over Recall: In SE tasks like library recommendation, it is often more important to be right about a sentiment than to find every instance of it. SEntiMoji excels here, showing high precision in identifying positive and negative sentiments, even in imbalanced datasets like the Java Library set.

Key Results

SEntiMoji consistently achieved the highest accuracy and F-scores across all four datasets, outperforming established tools like SentiStrength, SentiCR, and Senti4SD.

Performance comparison table

Statistical tests (McNemar’s test) confirmed that these improvements were significant. For instance, on the JIRA and Java Library datasets, SEntiMoji improved overall accuracy by more than 0.1 compared to the widely used SentiStrength.

Limitations and Future Work

Despite its success, SEntiMoji has a few "blind spots" identified through case studies:

  • Implicit Sentiment: The model struggles with subtle sarcasm or indirect blame (e.g., "I would recommend that you check the documentation for what they do").
  • Noisy Long Samples: In very long posts, the sheer volume of neutral information can sometimes drown out the sentiment signal.
  • Recall Levels: While precision is high, the authors acknowledge that more work is needed to improve recall—ensuring the model doesn't miss subtler emotional cues.

The authors suggest that this emoji-powered approach could be extended beyond basic sentiment to more complex tasks like detecting specific emotions (joy, anger, sadness) or measuring politeness in developer communities.

Summary updated

Abstract

Sentiment analysis has various application scenarios in software engineering (SE), such as detecting developers' emotions in commit messages and identifying their opinions on Q&A forums. However, commonly used out-of-the-box sentiment analysis tools cannot obtain reliable results on SE tasks and the misunderstanding of technical jargon is demonstrated to be the main reason. Then, researchers have to utilize labeled SE-related texts to customize sentiment analysis for SE tasks via a variety of algorithms. However, the scarce labeled data can cover only very limited expressions and thus cannot guarantee the analysis quality. To address such a problem, we turn to the easily available emoji usage data for help. More specifically, we employ emotional emojis as noisy labels of sentiments and propose a representation learning approach that uses both Tweets and GitHub posts containing emojis to learn sentiment-aware representations for SE-related texts. These emoji-labeled posts can not only supply the technical jargon, but also incorporate more general sentiment patterns shared across domains. They as well as labeled data are used to learn the final sentiment classifier. Compared to the existing sentiment analysis methods used in SE, the proposed approach can achieve significant improvement on representative benchmark datasets. By further contrast experiments, we find that the Tweets make a key contribution to the power of our approach. This finding informs future research not to unilaterally pursue the domain-specific resource, but try to transform knowledge from the open domain through ubiquitous signals such as emojis.
BibTeX
@inproceedings{chen2019sentimoji,
  author = {Zhenpeng Chen and Yanbin Cao and Xuan Lu and Qiaozhu Mei and Xuanzhe Liu},
  title = {{SEntiMoji: an emoji-powered learning approach for sentiment analysis in software engineering}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2019},
  url = {https://openreview.net/forum?id=5li5izF4Z1l},
}

Loading related content...

Figure