An Iterative Approach for Identifying Complaint Based Tweets in Social Media Platforms

Gyanesh Anand, Akash Kumar Gautam, Puneet Mathur, Debanjan Mahata, Rajiv Ratn Shah, Ramit Sawhney

Introduction

Social media platforms, particularly Twitter, serve as a vital source of real-time feedback for organizations in the transportation and logistics sectors. However, the high volume of data presents a significant challenge: extracting specific, actionable complaints from a sea of irrelevant noise. This task is complicated by the sparse and ungrammatical nature of "Twitter-speak," the rapid evolution of social media language, and the inherent data imbalance where neutral or objective posts far outnumber subjective grievances.

Methodology

To address these challenges, we propose a semi-supervised, iterative pipeline designed to handle continuous information streams. The process begins with Lexicon Curation, where industry-specific terms are identified from transport forums. Using tf-idf, an initial pool of "seed" phrases is generated to query Twitter.

The core of the approach is an Iterative Loop that allows the system to adapt to changing linguistic trends. After an initial manual annotation of a small subset, the system calculates a Domain Relevance Score for n-grams. This score identifies terms that appear significantly more frequently in informative tweets compared to a random background pool, effectively filtering out common noise. High-scoring terms are then added back as new seed phrases, creating a growing dictionary of complaint-related terms.

The iterative pipeline for seed phrase expansion and tweet classification.

Dataset and Feature Engineering

The study utilized a dataset of 19,300 tweets collected via transport lexicons, supplemented by 2,500 randomly sampled tweets to ensure lexical variety. For classification, a subset was annotated, resulting in 2,000 training tweets (702 relevant complaints) and 3,500 testing tweets.

The classification model, based on Logistic Regression with Elastic Net regularization, employed three categories of features:

  1. Linguistic Markers: Including Bag-of-Words (BOW), Part-of-Speech (POS) tags, and Word2Vec clusters.
  2. Sentiment Markers: Scores derived from MPQA, NRC, VADER, and Stanford lexicons.
  3. Information-Specific Markers: Handcrafted features such as "Intensifiers" (e.g., all-caps, repeated punctuation), politeness markers, and pronoun variations.

Results and Discussion

The performance of various feature sets was evaluated using 10-fold cross-validation. As shown in the table below, the Bag-of-Words (BOW) model emerged as the best overall performer, achieving an accuracy of 75.3% and an F1-score of 0.71.

Performance comparison across linguistic, sentiment, and information-specific markers.

A key finding was the effectiveness of Intensifiers. This feature, which tracks the "tone" of a tweet through capitalization and excessive punctuation, achieved 72.5% accuracy. This suggests that the stylistic expression of frustration is often as predictive as the literal content of the complaint. In terms of sentiment analysis, the Stanford model outperformed other lexicons, reaching 68.7% accuracy.

Conclusion

This iterative, semi-supervised approach provides a robust framework for identifying complaints in dynamic social media environments. By focusing on domain relevance rather than simple frequency, the model successfully isolates industry-specific pain points. While currently focused on the transport sector and utilizing traditional machine learning due to data constraints, the methodology is highly generalizable. Future work will explore the integration of demographic features and the scaling of this pipeline to multi-domain data streams.

Summary updated

Abstract

Twitter is a social media platform where users express opinions over a variety of issues. Posts offering grievances or complaints can be utilized by private/ public organizations to improve their service and promptly gauge a low-cost assessment. In this paper, we propose an iterative methodology which aims to identify complaint based posts pertaining to the transport domain. We perform comprehensive evaluations along with releasing a novel dataset for the research purposes.
BibTeX
@inproceedings{anand2020iterative,
  author = {Gyanesh Anand and Akash Kumar Gautam and Puneet Mathur and Debanjan Mahata and Rajiv Ratn Shah and Ramit Sawhney},
  title = {{An Iterative Approach for Identifying Complaint Based Tweets in Social Media Platforms}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7146},
}
Figure