Generate Adjective Sentiment Dictionary for Social Media Sentiment Analysis Using Constrained Nonnegative Matrix Factorization

Wei Peng, Dae Hoon ParkAAAI 2021 · DOI

Abstract

Sentiment analysis on social media faces significant hurdles due to informal language and the high cost of manual data labeling. This paper presents Constrained Symmetric Nonnegative Matrix Factorization (CSNMF), an automated method for generating high-coverage sentiment dictionaries. By integrating linguistic knowledge from WordNet with real-world usage patterns from Digg.com, the method produces a lexicon that outperforms traditional dictionaries. Experimental results demonstrate that this hybrid approach achieves superior precision and recall, effectively bridging the "informality gap" in social media sentiment analysis.

The Challenge of Social Media Sentiment

Traditional sentiment analysis tools often rely on lexicons like the General Inquirer (GI), which are designed for formal text. Social media, however, is characterized by slang, abbreviations, and evolving terminology (e.g., "diggable"). Machine learning models can address this but require massive, human-labeled datasets that are expensive to maintain. Lexicon-based approaches are more scalable but suffer from low coverage of informal terms. Furthermore, human agreement on the sentiment of social media comments is remarkably low—often around 44%—making manual labeling inherently difficult.

Methodology: The CSNMF Pipeline

The authors propose a multi-step pipeline to automatically expand a small seed of 52 known adjectives into a comprehensive lexicon:

  1. Seed Expansion: Initial positive and negative words are expanded using synonyms and antonyms from WordNet.
  2. Corpus Expansion: The system crawls a large social media corpus (Digg.com) to identify adjectives linked by conjunctions. Following linguistic principles, "and" typically links words of similar polarity, while "but" links opposites.
  3. CSNMF Algorithm: The core of the method is a semi-supervised clustering algorithm. It uses matrix factorization to group words into positive and negative clusters based on "attraction" (synonyms, "and" links) and "repulsion" (antonyms, "but" links). Crucially, the algorithm assigns a numerical sentiment strength score to each word, reflecting the confidence of its polarity assignment.

Experimental Setup and Datasets

The study utilized a 2GB corpus from Digg.com, containing over 1.8 million comments. To establish a ground truth for social media language, the authors used Amazon Mechanical Turk (AMT) to collect human labels for over 7,000 adjectives and 8,000 comments. This "social-media-specific" ground truth allowed for a more accurate evaluation of the generated dictionary compared to using formal benchmarks alone.

Key Results and Insights

The CSNMF method demonstrated significant improvements over standard clustering techniques and traditional lexicons.

  • Precision and Confidence: A key finding was the correlation between the algorithm's sentiment score and its accuracy. As shown in the performance evaluation, words with higher sentiment scores (representing higher confidence) yielded significantly better precision when compared against human labels.
Precision of generated sentiment dictionaries against human-labeled ground truth as a function of the number of top-ranked words.
  • Superior Coverage: The generated dictionary achieved a recall of 89.41%, catching far more sentiment-carrying words than the human-labeled AMT dictionary or the General Inquirer. In real-world classification tasks, this increased breadth proved more valuable than the "perfect" but limited labels of smaller manual dictionaries.
  • Hybrid Knowledge: The results confirm that combining "top-down" linguistic relationships (WordNet) with "bottom-up" usage data (Digg) is more effective than using either source in isolation.

Conclusion and Future Directions

The CSNMF approach provides a robust, automated way to keep sentiment dictionaries up-to-date with the rapidly changing language of social media. The effectiveness of this method is visualized in applications like "Social Sense Diffusion," which tracks opinion shifts over time.

A dashboard demonstrating real-world sentiment analysis on social media comments using the generated dictionary.

While the current focus on adjectives provides high precision, future work could expand the model to include nouns and verbs, and better address the complexities of sarcasm and context-dependent polarity.

Summary updated

Abstract

Although sentiment analysis has attracted a lot of research, little work has been done on social media data compared to product and movie reviews. This is due to the low accuracy that results from the more informal writing seen in social media data. Currently, most of sentiment analysis tools on social media choose the lexicon-based approach instead of the machine learning approach because the latter requires the huge challenge of obtaining enough human-labeled training data for extremely large-scale and diverse social opinion data. The lexicon-based approach requires a sentiment dictionary to determine opinion polarity. This dictionary can also provide useful features for any supervised learning method of the machine learning approach. However, many benchmark sentiment dictionaries do not cover the many informal and spoken words used in social media. In addition, they are not able to update frequently to include newly generated words online. In this paper, we present an automatic sentiment dictionary generation method, called Constrained Symmetric Nonnegative Matrix Factorization (CSNMF) algorithm, to assign polarity scores to each word in the dictionary, on a large social media corpus — digg.com. Moreover, we will demonstrate our study of Amazon Mechanical Turk (AMT) on social media word polarity, using both the human-labeled dictionaries from AMT and the General Inquirer Lexicon to compare our generated dictionary with. In our experiment, we show that combining links from both WordNet and the corpus to generate sentiment dictionaries does outperform using only one of them, and the words with higher sentiment scores yield better precision. Finally, we conducted a lexicon-based sentiment analysis on human-labeled social comments using our generated sentiment dictionary to show the effectiveness of our method.
BibTeX
@inproceedings{peng2021generate,
  author = {Wei Peng and Dae Hoon Park},
  title = {{Generate Adjective Sentiment Dictionary for Social Media Sentiment Analysis Using Constrained Nonnegative Matrix Factorization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v5i1.14090},
}
Figure