Abstract
Complaining is a basic speech act regularly used in human and computer-mediated communication to express a negative mismatch between reality and expectations. Automatically identifying complaints in social media is of utmost importance for organizations to improve customer experience and develop dialogue systems for handling grievances. This paper introduces the first systematic analysis of complaints in computational linguistics, utilizing a new annotated dataset of English tweets. We present an extensive linguistic analysis and train feature-based and neural models across nine domains, achieving a predictive performance of up to 79 F1 using distant supervision.
Defining the Complaint: Beyond Sentiment
In natural language processing, "complaints" are often conflated with "negative sentiment." However, linguistically, a complaint is defined specifically as a breach of expectations—a negative mismatch between reality and what a person expected from a product, service, or event.
As illustrated in the examples below, a tweet can express positive sentiment while still being a complaint, or express negative sentiment without being a formal complaint. For instance, a user might praise a brand while complaining about a specific policy change. Conversely, a simple insult like "You suck" lacks the context of breached expectations required for classification as a complaint.
Methodology and Dataset
The authors approach complaint detection as a binary text classification task. They introduce a brand new, publicly available dataset consisting of 3,449 annotated English tweets. These were sampled from 93 real-world customer service handles across nine domains, including Retail, Software, Transport, and Electronics. To ensure robustness, the dataset was balanced with random tweets and expertly annotated (Cohen’s Kappa = 0.731).
To overcome the limitations of small-scale manual annotation, the study also utilized distant supervision. They collected 18,218 tweets using complaint-adjacent hashtags (e.g., #badcustomerservice) and applied the EasyAdapt algorithm to adapt these noisy, out-of-domain signals to the target domains.
Linguistic Markers of Complaining
A key contribution of this work is the identification of specific linguistic features that distinguish complaints from other forms of discourse.
Several insights emerged from the feature analysis:
- The "My" vs. "You" Dynamic: Complaints heavily index on possessive pronouns followed by nouns (e.g., "my account," "my order"). They rarely use third-person pronouns, keeping the focus strictly on the user's personal experience and the responsible party.
- Questions over Exclamations: Surprisingly, exclamation marks are more indicative of non-complaints. Complainers often use question marks, framing grievances as inquiries (e.g., "Why is this not working?") to prompt a response while appearing less confrontational.
- Temporal References: Words like "still," "days," and "waiting" are massive linguistic giveaways. They provide the necessary context for the breached expectation, such as a delayed delivery or a long hold time.
- POS Patterns: Part-of-Speech (POS) tags like VBN (past participle) and VBZ (3rd person singular present) are highly correlated with complaints, often used to describe states of failure or ongoing issues.
Results and Domain Adaptation
The study found that standard sentiment models perform poorly on this task, achieving only 60 F1. In contrast, a Logistic Regression model using Bag-of-Words and POS tags achieved a 78.0 F1 score. By incorporating the 18k weakly labeled tweets via distant supervision, performance improved to 79.0 F1 (0.885 AUC).
Cross-domain experiments revealed that complaints vary significantly by industry. For example, Apparel complaints focus on "returns" and "stock," terms that do not transfer well to the Software domain. The results proved that formal domain adaptation techniques are necessary, as simply pooling data across domains can actually degrade performance.
Conclusion and Future Work
This research establishes a computational framework for identifying complaints as a distinct linguistic phenomenon. While the current models are highly effective at binary classification, future work will focus on identifying the specific target or aspect of the complaint and incorporating conversational context to improve accuracy in multi-tweet threads.
Summary updated