Analyzing the Intensity of Complaints on Social Media

Ming Fang, Shi Zong, Jing Li, Xinyu Dai, Shujian Huang, Jiajun Chen

Abstract

Complaining is a speech act that expresses a negative inconsistency between reality and human expectations. While prior studies mostly focus on identifying the existence or the type of complaints, in this work, we present the first study in computational linguistics of measuring the intensity of complaints from text. Analyzing complaints from such perspective is particularly useful, as complaints of certain degrees may cause severe consequences for companies or organizations. We create the first Chinese dataset containing 3,103 posts about complaints from Weibo, a popular Chinese social media platform. These posts are then annotated with complaints intensity scores using Best-Worst Scaling (BWS) method. We show that complaints intensity can be accurately estimated by computational models with the best mean square error achieving 0.11. Furthermore, we conduct a comprehensive linguistic analysis around complaints, including the connections between complaints and sentiment, and a cross-lingual comparison for complaints expressions used by Chinese and English speakers. We finally show that our complaints intensity scores can be incorporated for better estimating the popularity of posts on social media.

Introduction

While sentiment analysis has become a cornerstone of natural language processing, "complaining" represents a distinct and complex speech act. It is defined by the gap between a user's expectations and their perceived reality. Historically, NLP research has treated complaints as a binary classification problem—simply identifying if a post is a complaint or not—or categorized them into broad types like accusations.

However, the intensity of a complaint is often more critical than its mere existence. A mild expression of dissatisfaction requires a different organizational response than a high-intensity accusation of fraud. This study introduces a fine-grained approach to measuring complaint intensity, providing tools for organizations to prioritize urgent issues and manage reputational risks effectively.

Methodology and Data

The researchers developed a new dataset consisting of 3,103 Chinese posts from Weibo, specifically targeting the education sector. To ensure reliable intensity scores, they employed Best-Worst Scaling (BWS). Unlike traditional Likert scales, which can suffer from annotator bias, BWS requires annotators to choose the most and least intense complaints from small sets (4-tuples), resulting in a more stable continuous intensity score ranging from -1 to 1.

For modeling, the study evaluated several architectures, including SVR, Bi-LSTMs, and pre-trained Transformers like BERT, RoBERTa, and ERNIE. These models were fine-tuned using a Mean Square Error (MSE) loss objective to predict the continuous intensity scores.

Linguistic Insights: Complaint vs. Sentiment

A key finding of this research is that complaint intensity is not merely a proxy for negative sentiment. While they are correlated, the linguistic features driving each are distinct. Sentiment models often focus on emotional keywords, whereas complaint models prioritize the reasons behind the dissatisfaction.

Comparison of attention weights for sentiment vs. complaint intensity models.

Furthermore, the data revealed a "Halo Effect" in high-intensity complaints. As intensity increases, so does the length of the post. Furious users tend to expand their grievances, attacking multiple related aspects of the target rather than focusing on a single issue.

Cross-Lingual and Cultural Analysis

The study also compared Chinese Weibo complaints with English Twitter data. This analysis uncovered significant cultural differences in how complaints are articulated:

  • Indirectness: 80% of Chinese complaints were found to be indirect, compared to only 9% in English.
  • Irony: English speakers utilize irony much more frequently (26%) than Chinese speakers (10%) when complaining.

Predicting Post Popularity

Beyond linguistic analysis, the researchers demonstrated the practical utility of complaint intensity scores in predicting the virality of social media posts. By incorporating "complaint density" into popularity prediction models, they significantly reduced the Root Mean Square Error (RMSE) from 0.41 to 0.35.

Incorporating complaint intensity scores improves the accuracy of predicting post popularity over time.

Conclusion

This work establishes a framework for measuring the intensity of complaints on social media. The top-performing model, RoBERTa, achieved a Pearson correlation of 0.79, demonstrating that complaint severity can be accurately quantified. While challenges remain—particularly in detecting irony, which accounted for 43% of model errors—this research provides a vital new dimension for social media monitoring and organizational response strategies.

Summary updated

Abstract

Complaining is a speech act that expresses a negative inconsistency between reality and human expectations. While prior studies mostly focus on identifying the existence or the type of complaints, in this work, we present the first study in computational linguistics of measuring the intensity of complaints from text. Analyzing complaints from such perspective is particularly useful, as complaints of certain degrees may cause severe consequences for companies or organizations. We create the first Chinese dataset containing 3,103 posts about complaints from Weibo, a popular Chinese social media platform. These posts are then annotated with complaints intensity scores using Best-Worst Scaling (BWS) method. We show that complaints intensity can be accurately estimated by computational models with the best mean square error achieving 0.11. Furthermore, we conduct a comprehensive linguistic analysis around complaints, including the connections between complaints and sentiment, and a cross-lingual comparison for complaints expressions used by Chinese and English speakers. We finally show that our complaints intensity scores can be incorporated for better estimating the popularity of posts on social media.
BibTeX
@inproceedings{fang2022analyzing,
  author = {Ming Fang and Shi Zong and Jing Li and Xinyu Dai and Shujian Huang and Jiajun Chen},
  title = {{Analyzing the Intensity of Complaints on Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.132},
}
Figure