An Iterative Approach for Identifying Complaint Based Tweets in Social Media Platforms (Student Abstract)

Gyanesh Anand, Akash Kumar Gautam, Puneet Mathur, Debanjan Mahata, Rajiv Ratn Shah, Ramit Sawhney

Introduction

Social media platforms like Twitter offer a wealth of real-time feedback for transportation companies and urban planners. However, extracting actionable complaints from the vast volume of "noise" is a significant challenge. Traditional supervised learning methods often struggle due to the rapid evolution of language (slang, new infrastructure issues) and the inherent class imbalance where complaints represent a small fraction of total data. This paper presents a semi-supervised, iterative pipeline designed to identify transport-related complaints with minimal human intervention.

Methodology: The Iterative Pipeline

The proposed framework moves from specialized domain knowledge to general social media data through a multi-stage process. Initially, the researchers "bootstrapped" the system by scraping 500 posts from The Verge’s transportation forum. Using tf-idf, they identified 50 core "seed" terms to query Twitter, yielding an initial pool of 19,300 tweets.

A small subset of 1,500 tweets was manually annotated to identify "informative" complaints. These served as the basis for an iterative loop designed to refine the dataset.

The iterative pipeline for identifying transport-relevant tweets through seed phrase refinement and domain relevance scoring.

As shown in Figure 1, the loop involves extracting new seed phrases from informative tweets, filtering redundant content via cosine similarity, and calculating a Domain Relevance Score to ensure phrases remain contextually appropriate. This process was repeated over four iterations, eventually growing the transport-relevant dataset to 7,200 tweets.

Feature Analysis and Results

The researchers evaluated the effectiveness of various feature sets—Linguistic, Sentiment, and Information-specific—using Logistic Regression with Elastic Net regularization.

Performance comparison of different linguistic, sentiment, and information-specific markers in identifying complaint-based tweets.

The results, detailed in Figure 2, reveal several key insights:

  • Linguistic Superiority: Simple Bag-of-Words (BoW) features achieved the highest performance with 75.3% accuracy and a 0.71 F1-score. This suggests that specific "problem" keywords are more indicative of a complaint than complex emotional tone.
  • The Role of Intensifiers: Information-specific markers, particularly "Intensifiers" (e.g., excessive capitalization or punctuation), proved highly effective, reaching 72.5% accuracy.
  • Sentiment Limitations: While sentiment analysis (specifically the Stanford Sentiment model) provided a signal, it generally underperformed compared to linguistic and structural markers.

Conclusion and Future Work

This research demonstrates that a semi-supervised, iterative approach can effectively build a robust dataset for domain-specific complaint identification. By evolving the lexicon through multiple cycles, the system captures linguistic variations that static models miss.

Future work will explore the integration of neural networks, which may offer higher performance if larger labeled datasets can be generated. Additionally, the researchers plan to incorporate demographic analysis to better understand the populations affected by service issues and test the framework's generalizability in other sectors like healthcare and retail.

Summary updated

Abstract

Twitter is a social media platform where users express opinions over a variety of issues. Posts offering grievances or complaints can be utilized by private/ public organizations to improve their service and promptly gauge a low-cost assessment. In this paper, we propose an iterative methodology which aims to identify complaint based posts pertaining to the transport domain. We perform comprehensive evaluations along with releasing a novel dataset for the research purposes.
BibTeX
@inproceedings{anand2020iterative,
  author = {Gyanesh Anand and Akash Kumar Gautam and Puneet Mathur and Debanjan Mahata and Rajiv Ratn Shah and Ramit Sawhney},
  title = {{An Iterative Approach for Identifying Complaint Based Tweets in Social Media Platforms (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7146},
}
Figure