TL;DR
This paper introduces a novel semi-supervised iterative approach to automatically detect complaints about specific products or services on social media. By starting with a small set of labeled examples and iteratively refining linguistic indicators, the method effectively bootstraps a larger dataset of complaints, significantly outperforming initial manual labeling efforts.
What Problem Does This Paper Address?
Social media has become a crucial channel for users to express opinions about products and services, offering valuable insights for businesses. However, automatically identifying complaints within this vast, noisy data is challenging. Complaints are relatively rare, often embedded in informal language, and can be structurally varied due to rephrasing and redundancy. Traditional supervised methods require large, meticulously labeled datasets, which are expensive and time-consuming to create, especially for domain-specific complaints like those related to transportation services. This paper aims to bridge this gap by developing an efficient, semi-supervised method to discover and collect complaint-related social media posts.
Methods
The core of the paper is a semi-supervised iterative bootstrapping approach. It starts with a small, manually curated set of complaint examples (the "seed data"). This seed data is used to identify initial "linguistic indicators" – phrases or n-grams that strongly suggest a complaint. These indicators are then used to scan a larger, unannotated corpus of social media posts (in this case, Twitter) to find more potential complaints. These newly identified posts are added to the dataset, and the process repeats: new indicators are extracted from the expanded dataset, used to find even more complaints, and so on. This iterative cycle continues until no significant new indicators can be found, indicating convergence.
A key aspect of their method is the domain relevance criterion used for indicator extraction in later iterations. To avoid concept drift and ensure indicators are truly indicative of complaints, they measure an n-gram's relevance by comparing its frequency in the newly found complaint candidates to its frequency in a randomly sampled, unrelated dataset. This helps filter out general phrases that might appear frequently but aren't specific to complaints.
Datasets
- Seed Data (Ts): A manually annotated dataset of 326 transportation-related complaint tweets, initially collected from web forums and then filtered and annotated by human experts.
- Corpus C: A larger collection of 19,300 transportation-related tweets, obtained by querying Twitter for tweets containing phrases from an initial lexicon.
- Final Dataset (T): The dataset constructed through the iterative bootstrapping process, containing over 3,700 tweets.
- Evaluation Datasets:
- A random sample of 700 tweets from the final dataset was manually annotated to assess the quality of the collected complaints.
- Another set of 2,000 posts from the original corpus C was used for training machine learning models, with the 700 annotated tweets serving as the test set.
Key Insights and Ideas
- Bootstrapping for Rare Events: The iterative bootstrapping approach is highly effective for discovering rare events like complaints in large, noisy datasets. By leveraging initial weak signals, it can progressively build a substantial, relevant dataset.
- Domain Relevance is Crucial: Simply relying on high-frequency phrases isn't enough. A robust domain relevance measure is essential to filter out noise and prevent the indicator set from drifting away from the target concept (complaints).
- Complaints are Complex Speech Acts: The study highlights that identifying complaints involves more than just negative sentiment. Features related to requests, politeness, and specific linguistic structures also play a significant role.
- Simple Features Can Be Powerful: Surprisingly, basic unigram features (individual words) achieved the highest accuracy in their subsequent modeling experiments, suggesting that even simple linguistic cues can be highly predictive of complaints.
Key Results
- The iterative approach converged in four rounds, collecting 2,840 unique indicators and 3,732 tweets.
- Manual annotation of a random 700-tweet sample from the final dataset revealed that 47.4% were actual transportation complaints. This is a significant improvement over the initial seed data, where only 21.7% were complaints.
- In their modeling experiments, the best-performing model using unigram features achieved an accuracy of 75.3% and an F1-score of 0.70.
- Other features, including semantic, sentiment, orthographic, and request-related features, also showed predictive ability, with accuracies generally ranging from 67% to 72.5%.
Limitations and Future Work
- Domain Specificity: While the approach is designed for domain-specific detection, it requires an initial seed dataset and potentially domain-specific lexicons. Adapting it to entirely new domains might require significant effort.
- Manual Annotation Burden: Although the method reduces the need for extensive manual labeling, it still relies on an initial seed set and a final evaluation set that require human annotation.
- Future Work: The authors suggest applying this methodology to other products and services, exploring more sophisticated unsupervised or completely unsupervised techniques, and investigating its use in downstream applications like chatbots. They also note that complaint detection is related to other speech acts, implying further research into these connections.
Summary updated