Assessing Mobility Policy with AI-Driven Analysis of User-Generated Content

Floriano Tori, Charlotte van Vessem, Juliana Betancur Arenas, Vincent GinisOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

We recommend assessing mobility policies with AI-driven analysis of user-generated content. To achieve this policy recommendation, we propose to take into account the following: (1) Using large language models to analyse user-generated content is a reliable methodology for gathering and analysing typically overlooked relevant information regarding citizens' perceptions in the implementation of sustainable mobility policies; (2) The substantial processing capacity of these models, coupled with their ability to gather a great amount of information, enables decision-makers to supplement and enhance the often-limited traditional data collection methods. As exposed in the following case study, this methodology can provide historical perceptual information on transport modes, mobility policies, and infrastructure, among others; and (3) The ease of applying this methodology through AI open-source recent developments such as ChatGPT allows decision-makers and their teams to rapidly generate and assess a great amount of relevant data. This can facilitate policymakers' effectiveness and efficiency in the decision-making processes in urban mobility planning. However, policymakers should be aware of the characteristics of their selected population and use this as a complementary and evaluative method.

Introduction

Urban mobility is at a critical crossroads. While the EU’s "Green Deal" and "Fit for 55" initiatives aim for climate neutrality, implementing these policies often triggers significant public backlash, such as protests against low-traffic neighborhoods. Traditional methods for gauging public opinion—manual surveys and traffic counts—are slow, expensive, and often fail to capture the "why" behind citizen frustration. They frequently miss the nuances of diverse groups and cannot provide the real-time feedback loop needed for agile policy adjustment. This paper explores whether AI-driven analysis of social media can serve as a reliable, scalable "thermometer" for public sentiment.

Methodology

The researchers developed an interdisciplinary pipeline combining data analytics with mobility behavior studies. They scraped Twitter (now X) for mobility-related content in the Brussels Capital Region from 2017 to 2022, capturing 15,001 tweets. To ensure the AI was reliable, three human experts manually labeled a random sample of 500 tweets across three categories: Mode, Domain, and Sentiment. GPT-4 was then tasked with labeling the same tweets to benchmark its performance.

Comparison of annotations between three human experts (SSH and STEM) and GPT-4 across Mode, Domain, and Sentiment categories.

The team used the Cohen’s Kappa coefficient to measure agreement. The results showed that GPT-4’s performance was essentially the same as the agreement between the human experts themselves, validating the LLM as a peer to expert annotators.

Key Results

Once validated, GPT-4 was used to analyze the broader dataset. The study processed 15,001 tweets, eventually narrowing down to 5,059 high-confidence labels where the AI could clearly identify the mode, domain, and sentiment.

Distribution of transport modes, policy domains, and sentiments across the full analyzed dataset.
  • Dominant Topics: Active mobility (bikes/e-bikes) received the most attention (over 2,000 tweets), followed by motorized mobility and public transport.
  • Infrastructure Focus: Across all transport modes, the majority of public discourse centered on infrastructure, followed by policy and operations.
  • Sentiment Trends: A significant portion of the discourse was categorized as negative; however, qualitative analysis suggests this often reflects a desire for better or safer infrastructure rather than opposition to sustainable goals.

Discussion and Insights

The findings suggest that LLMs can augment or replace expensive human coding for large-scale qualitative research. However, technical hurdles remain. Agreement scores were higher for "Parent Tweets" (0.61) than for "Reply Tweets" (0.54), highlighting that LLMs struggle to interpret sentiment when the original context is fragmented.

Furthermore, spikes in Twitter activity correlated directly with major policy milestones, such as the implementation of the Brussels Sustainable Urban Mobility Plan (SUMP) in late 2022. This demonstrates the potential for real-time policy monitoring.

Limitations and Future Work

While powerful, this methodology has limitations. Twitter users are not a perfect proxy for the general population, and digital divides mean certain demographics (the elderly, lower-income groups) may be underrepresented. Future research should apply this model to other platforms beyond Twitter to capture a broader demographic slice and refine the model's ability to handle linguistic nuances like sarcasm and irony.

Summary updated

Abstract

Abstract We recommend assessing mobility policies with AI-driven analysis of user-generated content. To achieve this policy recommendation, we propose to take into account the following: (1) Using large language models to analyse user-generated content is a reliable methodology for gathering and analysing typically overlooked relevant information regarding citizens' perceptions in the implementation of sustainable mobility policies; (2) The substantial processing capacity of these models, coupled with their ability to gather a great amount of information, enables decision-makers to supplement and enhance the often-limited traditional data collection methods. As exposed in the following case study, this methodology can provide historical perceptual information on transport modes, mobility policies, and infrastructure, among others; and (3) The ease of applying this methodology through AI open-source recent developments such as ChatGPT allows decision-makers and their teams to rapidly generate and assess a great amount of relevant data. This can facilitate policymakers' effectiveness and efficiency in the decision-making processes in urban mobility planning. However, policymakers should be aware of the characteristics of their selected population and use this as a complementary and evaluative method.
BibTeX
@inproceedings{tori2024assessing,
  author = {Floriano Tori and Charlotte van Vessem and Juliana Betancur Arenas and Vincent Ginis},
  title = {{Assessing Mobility Policy with AI-Driven Analysis of User-Generated Content}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.1007/978-3-031-67936-0\_8},
}
Figure