From Twitter to Reasoner: Understand Mobility Travel Modes and Sentiment Using Large Language Models

Kangrui Ruan, Xinyang Wang, Xuan Di

Introduction

Urban planners and transit operators increasingly rely on real-time feedback to optimize city infrastructure. Social media platforms, particularly Twitter (X), offer a vast repository of public sentiment regarding transportation. However, extracting meaningful insights from this data is challenging due to its unstructured nature and the ambiguity of natural language. Traditional methods requiring extensive manual labeling are slow and often fail to capture the nuance of travel modes and specific pain points. This paper presents a novel framework leveraging Large Language Models (LLMs) to automate the identification of travel modes and sentiment analysis without the need for manual data labeling.

Methodology: The Reasoner-Verifier Framework

The core of the proposed system is a two-stage "Reasoner-Verifier" architecture designed to process "messy" social media data into structured insights.

The Reasoner-Verifier framework for processing social media data.
  1. The Reasoner: This component utilizes an LLM to analyze pre-processed tweets. It performs three simultaneous tasks: identifying the travel mode (e.g., Subway, Bus, Bike), determining the sentiment (Positive, Neutral, Negative), and providing a "reasoning check" to justify its classification.
  2. The Verifier: To maintain high data quality, a second stage involves verification. This can be performed by a more advanced LLM (such as GPT-4) or a human evaluator to confirm the Reasoner's output.

The study evaluated three models—GPT-3.5, Llama2-7B, and Mistral-7B—using various prompting strategies, including Instruction-following, In-Context Learning (ICL), Chain-of-Thought (CoT), and Analogical Prompting.

Data and Experiments

The framework was tested on a dataset of over 250,000 tweets collected in New York City (NYC) between 2020 and 2022. NYC was selected due to its diverse transportation landscape. Initial data collection used keywords like "MTA" and "subway," but the framework was responsible for filtering out irrelevant noise (e.g., distinguishing between the "Subway" transit system and the sandwich shop).

Results and Discussion

The experimental results highlighted several key findings:

  • Model Performance: GPT-3.5 outperformed open-source alternatives, achieving verification scores of 0.82 for travel mode identification and 0.75 for sentiment analysis.
  • Prompting Effectiveness: In-Context Learning (ICL) proved to be the most effective strategy for these specific classification tasks, balancing accuracy and reasoning.
  • Sentiment Trends: A prevailing negative bias was observed across social media posts, confirming that users are more likely to "vent" about service failures than praise successes. Public transit modes (Subway and Bus) faced more criticism regarding delays and safety, while Bicycles and Ride-sourcing (Taxi/Uber) showed relatively higher satisfaction levels.
  • Urban Insights: Biking emerged as a significant topic of discussion in NYC, with users frequently advocating for better infrastructure and secure parking.

Conclusion

The Reasoner-Verifier framework demonstrates that LLMs can effectively replace manual labeling for large-scale urban mobility studies. While the current study is focused on NYC, the methodology is scalable to other urban environments. Future research will focus on integrating real-time monitoring capabilities to assist transit authorities in responding dynamically to public feedback.

Summary updated

BibTeX
@inproceedings{ruan2024from,
  author = {Kangrui Ruan and Xinyang Wang and Xuan Di},
  title = {{From Twitter to Reasoner: Understand Mobility Travel Modes and Sentiment Using Large Language Models}},
  booktitle = {ITSC},
  year = {2024},
  doi = {10.1109/itsc58415.2024.10919612},
}
Figure