Authors: Daniel Preoţiuc-Pietro, Mihaela Gaman, Nikolaos Aletras
Source: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019)
Abstract
This study presents the first systematic computational analysis of "complaining" as a distinct speech act on social media. Moving beyond traditional sentiment analysis, the researchers developed a new annotated dataset and predictive models capable of identifying complaints with up to 79% F1-score, demonstrating that complaints can be accurately detected even when the language is not explicitly "angry."
Introduction: Defining the Complaint
In Natural Language Processing (NLP), negative feedback is often broadly categorized as "negative sentiment." However, for organizations managing customer relations, there is a critical distinction between a general expression of dislike and a specific complaint. The authors define a complaint as a "breach of expectations."
While a phrase like "This movie sucks" expresses negative sentiment, a statement like "I ordered this a week ago and it still hasn't arrived" is a complaint requiring resolution. As shown in Figure 1, these categories often overlap but are fundamentally different; a user can complain politely or express frustration without a specific grievance.
Methodology and Data
To address this, the researchers introduced a new dataset of 3,449 tweets directed at 93 corporate handles across nine domains, including Retail, Transport, and Electronics. Each tweet was manually annotated by experts as either a "Complaint" or "Not Complaint."
The study compared several modeling approaches:
- Feature-based Logistic Regression: Utilizing linguistic features like unigrams and Part-of-Speech (POS) tags.
- Neural Architectures: Including Multi-Layer Perceptrons (MLP) and Long Short-Term Memory (LSTM) networks.
- Distant Supervision: Leveraging 18,000 tweets with complaint-related hashtags (e.g., #badservice) to augment the training data via the EasyAdapt algorithm.
Linguistic Indicators of Complaining
The analysis revealed specific linguistic markers that distinguish complaints from general negative sentiment.
Key findings include:
- Personalization: Complaints are highly personal, frequently using possessive pronouns like "my" (e.g., "my account," "my order").
- Temporal Context: Words like "still" and "days" are strong indicators, highlighting the duration of the service failure.
- Grammatical Patterns: Complaints often use past participles (VBN) to describe past events that led to the current issue.
- Interrogative Structure: Many complaints are phrased as questions ("Why is this not working?") rather than direct statements. Interestingly, question marks are more predictive of complaints than exclamation marks, which are more common in general emotional outbursts.
Results and Performance
The best-performing model—Logistic Regression combined with Distant Supervision—achieved an F1-score of 79.0 and an AUC of 0.885. Notably, simple Bag-of-Words features (77.5 F1) significantly outperformed sentiment-only models (60.0 F1), proving that identifying the act of complaining requires more than just detecting "bad" words.
While neural models (LSTM/MLP) performed competitively, they did not surpass the feature-engineered linear models. This suggests that for datasets of this size, explicit linguistic cues remain highly reliable indicators of user intent.
Conclusion and Future Directions
This research provides a foundation for automated customer service systems to better prioritize and respond to actionable feedback. Future work aims to move toward Aspect-Based Complaint Identification, identifying not just that a complaint exists, but the specific product feature or service stage involved. Expanding this analysis to longer-form text and conversational threads could further improve detection accuracy in diverse communication channels.
Summary updated