Introduction
Social media platforms, particularly Twitter, serve as a vital source of real-time feedback for organizations in the transportation and logistics sectors. However, the high volume of data presents a significant challenge: extracting specific, actionable complaints from a sea of irrelevant noise. This task is complicated by the sparse and ungrammatical nature of "Twitter-speak," the rapid evolution of social media language, and the inherent data imbalance where neutral or objective posts far outnumber subjective grievances.
Methodology
To address these challenges, we propose a semi-supervised, iterative pipeline designed to handle continuous information streams. The process begins with Lexicon Curation, where industry-specific terms are identified from transport forums. Using tf-idf, an initial pool of "seed" phrases is generated to query Twitter.
The core of the approach is an Iterative Loop that allows the system to adapt to changing linguistic trends. After an initial manual annotation of a small subset, the system calculates a Domain Relevance Score for n-grams. This score identifies terms that appear significantly more frequently in informative tweets compared to a random background pool, effectively filtering out common noise. High-scoring terms are then added back as new seed phrases, creating a growing dictionary of complaint-related terms.
Dataset and Feature Engineering
The study utilized a dataset of 19,300 tweets collected via transport lexicons, supplemented by 2,500 randomly sampled tweets to ensure lexical variety. For classification, a subset was annotated, resulting in 2,000 training tweets (702 relevant complaints) and 3,500 testing tweets.
The classification model, based on Logistic Regression with Elastic Net regularization, employed three categories of features:
- Linguistic Markers: Including Bag-of-Words (BOW), Part-of-Speech (POS) tags, and Word2Vec clusters.
- Sentiment Markers: Scores derived from MPQA, NRC, VADER, and Stanford lexicons.
- Information-Specific Markers: Handcrafted features such as "Intensifiers" (e.g., all-caps, repeated punctuation), politeness markers, and pronoun variations.
Results and Discussion
The performance of various feature sets was evaluated using 10-fold cross-validation. As shown in the table below, the Bag-of-Words (BOW) model emerged as the best overall performer, achieving an accuracy of 75.3% and an F1-score of 0.71.
A key finding was the effectiveness of Intensifiers. This feature, which tracks the "tone" of a tweet through capitalization and excessive punctuation, achieved 72.5% accuracy. This suggests that the stylistic expression of frustration is often as predictive as the literal content of the complaint. In terms of sentiment analysis, the Stanford model outperformed other lexicons, reaching 68.7% accuracy.
Conclusion
This iterative, semi-supervised approach provides a robust framework for identifying complaints in dynamic social media environments. By focusing on domain relevance rather than simple frequency, the model successfully isolates industry-specific pain points. While currently focused on the transport sector and utilizing traditional machine learning due to data constraints, the methodology is highly generalizable. Future work will explore the integration of demographic features and the scaling of this pipeline to multi-domain data streams.
Summary updated