Analyzing the Intensity of Complaints on Social Media

Ming Fang, Shi Zong, Jing Li, Xinyu Dai, Shujian Huang, Jiajun Chen

Abstract

This study introduces a novel computational linguistics framework to quantify the intensity of complaints on social media. While traditional sentiment analysis distinguishes between positive and negative emotions, it often fails to capture the nuanced spectrum of grievance severity. By developing a dataset of over 3,000 Chinese Weibo posts and utilizing transformer-based models, this research demonstrates that complaint intensity is a distinct metric from general sentiment and serves as a critical indicator for predicting topic virality.

Introduction

In the landscape of social media monitoring, identifying a complaint is only the first step. There is a significant difference between a minor grievance, such as "the coffee was lukewarm," and a severe accusation like "this company is a fraudulent scam." These represent vastly different levels of reputational risk. Prior research has categorized complaints into types (e.g., accusation, disapproval) but has largely ignored the intensity within those categories. This paper addresses this gap, arguing that measuring intensity is essential for organizations to prioritize feedback and understand the "halo effect"—where a single negative experience triggers a broader critique of a brand.

Methodology

The researchers employed three primary strategies to analyze complaint intensity:

  1. Annotation via Best-Worst Scaling (BWS): To avoid the inconsistencies of traditional 1-5 Likert scales, the study used BWS. Annotators compared sets of four posts to identify the least and most intense complaints, resulting in stable intensity scores ranging from -1 to 1.
  2. Predictive Modeling: The study benchmarked several architectures, including Support Vector Regression (SVR), Bidirectional LSTMs, and pre-trained transformers such as BERT, RoBERTa, and ERNIE.
  3. Cross-Lingual Analysis: The authors compared Chinese Weibo posts with English Twitter datasets to identify cultural differences in how grievances are expressed, focusing on directness and the use of irony.

Linguistic Insights: Complaints vs. Sentiment

A key finding is that complaint intensity is not synonymous with negative sentiment. While they correlate, the underlying linguistic features differ. Sentiment models typically focus on emotional keywords, whereas complaint models prioritize the specific reasons for the grievance.

Figure 4: Comparison of attention maps for sentiment vs. complaint intensity models. Sentiment models focus on emotional cues, while complaint models focus on the factual triggers of the grievance.

Furthermore, the study identified a "halo effect" in text: stronger complaints are generally longer and more detailed. Highly dissatisfied users tend to expand their critique from a primary issue to peripheral targets, such as attacking support staff or unrelated services.

Results and Impact

The RoBERTa model emerged as the most effective for predicting intensity, achieving a Pearson’s correlation of 0.79. Beyond classification, the practical utility of measuring intensity was demonstrated in popularity prediction. By incorporating "complaint density"—the ratio of high-intensity complaints to total posts—into a baseline model, the researchers significantly improved the accuracy of predicting which topics would go viral.

Figure 5: Popularity prediction performance. The model incorporating complaint intensity (Predict) tracks the real-world popularity (Real) more closely than the baseline model.

The inclusion of intensity features reduced the Root Mean Square Error (RMSE) from 0.41 to 0.35, proving that intense early complaints are a reliable early-warning signal for trending topics.

Conclusion

This research establishes complaint intensity as a vital metric for social media analysis. While models still face challenges with irony and implicit dissatisfaction, the ability to quantify the severity of grievances offers profound benefits for crisis management and public sentiment tracking. Future work will focus on expanding the dataset across more diverse domains, such as e-commerce and public utilities, to further refine these predictive capabilities.

Summary updated

Abstract

Complaining is a speech act that expresses a negative inconsistency between reality and human expectations. While prior studies mostly focus on identifying the existence or the type of complaints, in this work, we present the first study in computational linguistics of measuring the intensity of complaints from text. Analyzing complaints from such perspective is particularly useful, as complaints of certain degrees may cause severe consequences for companies or organizations. We create the first Chinese dataset containing 3,103 posts about complaints from Weibo, a popular Chinese social media platform. These posts are then annotated with complaints intensity scores using Best-Worst Scaling (BWS) method. We show that complaints intensity can be accurately estimated by computational models with the best mean square error achieving 0.11. Furthermore, we conduct a comprehensive linguistic analysis around complaints, including the connections between complaints and sentiment, and a cross-lingual comparison for complaints expressions used by Chinese and English speakers. We finally show that our complaints intensity scores can be incorporated for better estimating the popularity of posts on social media.
BibTeX
@inproceedings{fang2022analyzing,
  author = {Ming Fang and Shi Zong and Jing Li and Xinyu Dai and Shujian Huang and Jiajun Chen},
  title = {{Analyzing the Intensity of Complaints on Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.132},
}
Figure