Abstract
We recommend assessing mobility policies with AI-driven analysis of user-generated content. To achieve this policy recommendation, we propose to take into account the following: (1) Using large language models to analyse user-generated content is a reliable methodology for gathering and analysing typically overlooked relevant information regarding citizens' perceptions in the implementation of sustainable mobility policies; (2) The substantial processing capacity of these models, coupled with their ability to gather a great amount of information, enables decision-makers to supplement and enhance the often-limited traditional data collection methods. As exposed in the following case study, this methodology can provide historical perceptual information on transport modes, mobility policies, and infrastructure, among others; and (3) The ease of applying this methodology through AI open-source recent developments such as ChatGPT allows decision-makers and their teams to rapidly generate and assess a great amount of relevant data. This can facilitate policymakers' effectiveness and efficiency in the decision-making processes in urban mobility planning. However, policymakers should be aware of the characteristics of their selected population and use this as a complementary and evaluative method.
Introduction
Urban mobility is at a critical crossroads. While the EU’s "Green Deal" and "Fit for 55" initiatives aim for climate neutrality, implementing these policies often triggers significant public backlash, such as protests against low-traffic neighborhoods. Traditional methods for gauging public opinion—manual surveys and traffic counts—are slow, expensive, and often fail to capture the "why" behind citizen frustration. They frequently miss the nuances of diverse groups and cannot provide the real-time feedback loop needed for agile policy adjustment. This paper explores whether AI-driven analysis of social media can serve as a reliable, scalable "thermometer" for public sentiment.
Methodology
The researchers developed an interdisciplinary pipeline combining data analytics with mobility behavior studies. They scraped Twitter (now X) for mobility-related content in the Brussels Capital Region from 2017 to 2022, capturing 15,001 tweets. To ensure the AI was reliable, three human experts manually labeled a random sample of 500 tweets across three categories: Mode, Domain, and Sentiment. GPT-4 was then tasked with labeling the same tweets to benchmark its performance.
The team used the Cohen’s Kappa coefficient to measure agreement. The results showed that GPT-4’s performance was essentially the same as the agreement between the human experts themselves, validating the LLM as a peer to expert annotators.
Key Results
Once validated, GPT-4 was used to analyze the broader dataset. The study processed 15,001 tweets, eventually narrowing down to 5,059 high-confidence labels where the AI could clearly identify the mode, domain, and sentiment.
- Dominant Topics: Active mobility (bikes/e-bikes) received the most attention (over 2,000 tweets), followed by motorized mobility and public transport.
- Infrastructure Focus: Across all transport modes, the majority of public discourse centered on infrastructure, followed by policy and operations.
- Sentiment Trends: A significant portion of the discourse was categorized as negative; however, qualitative analysis suggests this often reflects a desire for better or safer infrastructure rather than opposition to sustainable goals.
Discussion and Insights
The findings suggest that LLMs can augment or replace expensive human coding for large-scale qualitative research. However, technical hurdles remain. Agreement scores were higher for "Parent Tweets" (0.61) than for "Reply Tweets" (0.54), highlighting that LLMs struggle to interpret sentiment when the original context is fragmented.
Furthermore, spikes in Twitter activity correlated directly with major policy milestones, such as the implementation of the Brussels Sustainable Urban Mobility Plan (SUMP) in late 2022. This demonstrates the potential for real-time policy monitoring.
Limitations and Future Work
While powerful, this methodology has limitations. Twitter users are not a perfect proxy for the general population, and digital divides mean certain demographics (the elderly, lower-income groups) may be underrepresented. Future research should apply this model to other platforms beyond Twitter to capture a broader demographic slice and refine the model's ability to handle linguistic nuances like sarcasm and irony.
Summary updated