Automatically Identifying Complaints in Social Media

Daniel Preotiuc-Pietro, Mihaela Gaman, Nikolaos Aletras

Authors: Daniel Preoţiuc-Pietro, Mihaela Gaman, Nikolaos Aletras
Source: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019)

Abstract

This study presents the first systematic computational analysis of "complaining" as a distinct speech act on social media. Moving beyond traditional sentiment analysis, the researchers developed a new annotated dataset and predictive models capable of identifying complaints with up to 79% F1-score, demonstrating that complaints can be accurately detected even when the language is not explicitly "angry."

Introduction: Defining the Complaint

In Natural Language Processing (NLP), negative feedback is often broadly categorized as "negative sentiment." However, for organizations managing customer relations, there is a critical distinction between a general expression of dislike and a specific complaint. The authors define a complaint as a "breach of expectations."

While a phrase like "This movie sucks" expresses negative sentiment, a statement like "I ordered this a week ago and it still hasn't arrived" is a complaint requiring resolution. As shown in Figure 1, these categories often overlap but are fundamentally different; a user can complain politely or express frustration without a specific grievance.

Examples of tweets labeled for complaints (C) and negative sentiment (S), illustrating that the two categories are distinct.

Methodology and Data

To address this, the researchers introduced a new dataset of 3,449 tweets directed at 93 corporate handles across nine domains, including Retail, Transport, and Electronics. Each tweet was manually annotated by experts as either a "Complaint" or "Not Complaint."

The study compared several modeling approaches:

  • Feature-based Logistic Regression: Utilizing linguistic features like unigrams and Part-of-Speech (POS) tags.
  • Neural Architectures: Including Multi-Layer Perceptrons (MLP) and Long Short-Term Memory (LSTM) networks.
  • Distant Supervision: Leveraging 18,000 tweets with complaint-related hashtags (e.g., #badservice) to augment the training data via the EasyAdapt algorithm.

Linguistic Indicators of Complaining

The analysis revealed specific linguistic markers that distinguish complaints from general negative sentiment.

Unigram and POS features most strongly correlated with complaints versus non-complaints.

Key findings include:

  • Personalization: Complaints are highly personal, frequently using possessive pronouns like "my" (e.g., "my account," "my order").
  • Temporal Context: Words like "still" and "days" are strong indicators, highlighting the duration of the service failure.
  • Grammatical Patterns: Complaints often use past participles (VBN) to describe past events that led to the current issue.
  • Interrogative Structure: Many complaints are phrased as questions ("Why is this not working?") rather than direct statements. Interestingly, question marks are more predictive of complaints than exclamation marks, which are more common in general emotional outbursts.

Results and Performance

The best-performing model—Logistic Regression combined with Distant Supervision—achieved an F1-score of 79.0 and an AUC of 0.885. Notably, simple Bag-of-Words features (77.5 F1) significantly outperformed sentiment-only models (60.0 F1), proving that identifying the act of complaining requires more than just detecting "bad" words.

While neural models (LSTM/MLP) performed competitively, they did not surpass the feature-engineered linear models. This suggests that for datasets of this size, explicit linguistic cues remain highly reliable indicators of user intent.

Conclusion and Future Directions

This research provides a foundation for automated customer service systems to better prioritize and respond to actionable feedback. Future work aims to move toward Aspect-Based Complaint Identification, identifying not just that a complaint exists, but the specific product feature or service stage involved. Expanding this analysis to longer-form text and conversational threads could further improve detection accuracy in diverse communication channels.

Summary updated

BibTeX
@inproceedings{preotiucpietro2019automatically,
  author = {Daniel Preotiuc-Pietro and Mihaela Gaman and Nikolaos Aletras},
  title = {{Automatically Identifying Complaints in Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=cVuq2zYVakW},
}
Figure