Semi-Supervised Iterative Approach for Domain-Specific Complaint Detection in Social Media

Akash Gautam, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah2020 · DOI

TL;DR

This paper introduces a novel semi-supervised iterative approach to automatically detect complaints about specific products or services on social media. By starting with a small set of labeled examples and iteratively refining linguistic indicators, the method effectively bootstraps a larger dataset of complaints, significantly outperforming initial manual labeling efforts.

What Problem Does This Paper Address?

Social media has become a crucial channel for users to express opinions about products and services, offering valuable insights for businesses. However, automatically identifying complaints within this vast, noisy data is challenging. Complaints are relatively rare, often embedded in informal language, and can be structurally varied due to rephrasing and redundancy. Traditional supervised methods require large, meticulously labeled datasets, which are expensive and time-consuming to create, especially for domain-specific complaints like those related to transportation services. This paper aims to bridge this gap by developing an efficient, semi-supervised method to discover and collect complaint-related social media posts.

Methods

The core of the paper is a semi-supervised iterative bootstrapping approach. It starts with a small, manually curated set of complaint examples (the "seed data"). This seed data is used to identify initial "linguistic indicators" – phrases or n-grams that strongly suggest a complaint. These indicators are then used to scan a larger, unannotated corpus of social media posts (in this case, Twitter) to find more potential complaints. These newly identified posts are added to the dataset, and the process repeats: new indicators are extracted from the expanded dataset, used to find even more complaints, and so on. This iterative cycle continues until no significant new indicators can be found, indicating convergence.

A key aspect of their method is the domain relevance criterion used for indicator extraction in later iterations. To avoid concept drift and ensure indicators are truly indicative of complaints, they measure an n-gram's relevance by comparing its frequency in the newly found complaint candidates to its frequency in a randomly sampled, unrelated dataset. This helps filter out general phrases that might appear frequently but aren't specific to complaints.

Datasets

  • Seed Data (Ts): A manually annotated dataset of 326 transportation-related complaint tweets, initially collected from web forums and then filtered and annotated by human experts.
  • Corpus C: A larger collection of 19,300 transportation-related tweets, obtained by querying Twitter for tweets containing phrases from an initial lexicon.
  • Final Dataset (T): The dataset constructed through the iterative bootstrapping process, containing over 3,700 tweets.
  • Evaluation Datasets:
    • A random sample of 700 tweets from the final dataset was manually annotated to assess the quality of the collected complaints.
  • Another set of 2,000 posts from the original corpus C was used for training machine learning models, with the 700 annotated tweets serving as the test set.

Key Insights and Ideas

  • Bootstrapping for Rare Events: The iterative bootstrapping approach is highly effective for discovering rare events like complaints in large, noisy datasets. By leveraging initial weak signals, it can progressively build a substantial, relevant dataset.
  • Domain Relevance is Crucial: Simply relying on high-frequency phrases isn't enough. A robust domain relevance measure is essential to filter out noise and prevent the indicator set from drifting away from the target concept (complaints).
  • Complaints are Complex Speech Acts: The study highlights that identifying complaints involves more than just negative sentiment. Features related to requests, politeness, and specific linguistic structures also play a significant role.
  • Simple Features Can Be Powerful: Surprisingly, basic unigram features (individual words) achieved the highest accuracy in their subsequent modeling experiments, suggesting that even simple linguistic cues can be highly predictive of complaints.

Key Results

  • The iterative approach converged in four rounds, collecting 2,840 unique indicators and 3,732 tweets.
  • Manual annotation of a random 700-tweet sample from the final dataset revealed that 47.4% were actual transportation complaints. This is a significant improvement over the initial seed data, where only 21.7% were complaints.
  • In their modeling experiments, the best-performing model using unigram features achieved an accuracy of 75.3% and an F1-score of 0.70.
  • Other features, including semantic, sentiment, orthographic, and request-related features, also showed predictive ability, with accuracies generally ranging from 67% to 72.5%.

Limitations and Future Work

  • Domain Specificity: While the approach is designed for domain-specific detection, it requires an initial seed dataset and potentially domain-specific lexicons. Adapting it to entirely new domains might require significant effort.
  • Manual Annotation Burden: Although the method reduces the need for extensive manual labeling, it still relies on an initial seed set and a final evaluation set that require human annotation.
  • Future Work: The authors suggest applying this methodology to other products and services, exploring more sophisticated unsupervised or completely unsupervised techniques, and investigating its use in downstream applications like chatbots. They also note that complaint detection is related to other speech acts, implying further research into these connections.

Summary updated

Abstract

In this paper, we present a semi-supervised bootstrapping approach to detect product or service related complaints in social media. Our approach begins with a small collection of annotated samples which are used to identify a preliminary set of linguistic indicators pertinent to complaints. These indicators are then used to expand the dataset. The expanded dataset is again used to extract more indicators. This process is applied for several iterations until we can no longer find any new indicators. We evaluated this approach on a Twitter corpus specifically to detect complaints about transportation services. We started with an annotated set of 326 samples of transportation complaints, and after four iterations of the approach, we collected 2,840 indicators and over 3,700 tweets. We annotated a random sample of 700 tweets from the final dataset and observed that nearly half the samples were actual transportation complaints. Lastly, we also studied how different features based on semantics, orthographic properties, and sentiment contribute towards the prediction of complaints.
BibTeX
@misc{gautam2020semi,
  author = {Akash Gautam and Debanjan Mahata and Rakesh Gosangi and Rajiv Ratn Shah},
  title = {{Semi-Supervised Iterative Approach for Domain-Specific Complaint Detection in Social Media}},
  year = {2020},
  doi = {10.18653/v1/2020.ecnlp-1.7},
}

Loading related content...

Figure