TL;DR
CityBench is the first systematic, interactive benchmark designed to evaluate Large Language Models (LLMs) and Vision-Language Models (VLMs) across a diverse range of urban tasks. By testing 30 models in 13 global cities, the researchers found that while current models excel at semantic understanding, they struggle significantly with domain-specific numerical reasoning and complex urban decision-making.
What Problem Does This Paper Address?
As we rush to integrate LLMs into everything from urban planning to autonomous traffic management, a critical question remains: how capable are these models in the messy, dynamic, and multi-modal reality of a city?
Existing evaluations for LLMs in the urban domain are often fragmented—focusing on a single city, a single data type (like just text or just satellite images), or static tasks that don't account for the interactive nature of urban environments. The challenge lies in the sheer diversity of urban data and the highly dynamic nature of city life. This paper introduces CityBench to bridge this gap, providing a scalable and systematic platform to see if "world models" actually understand the urban world.
Methods
The authors developed a three-tiered framework to move beyond static question-answering:
- CityData: A massive multi-source data collector that integrates geospatial data (OpenStreetMap), visual data (Google/Baidu street views and Esri satellite imagery), and human activity data (Foursquare check-ins and synthetic movement patterns).
- CitySimu: An interactive simulation engine. Unlike static benchmarks, CitySimu allows models to act as agents. It simulates individual mobility, visual environments, and microscopic traffic dynamics, providing APIs for models to "sense" the environment and "act" (e.g., changing a traffic signal).
- CityBench: The evaluation suite itself, comprising 8 representative tasks split into two categories: Perception & Understanding (like geolocalization and infrastructure inference) and Planning & Decision-Making (like traffic signal control and urban exploration).
Datasets
The benchmark utilizes CityData, which covers 13 major global cities including New York, London, Tokyo, Nairobi, and Shanghai. The scale is impressive:
- 20K+ images (satellite and street view).
- 30K+ questions generated through a mix of templates and LLM-assisted methods.
- Human activity data from Foursquare and synthetic OD (Origin-Destination) matrices to power realistic movement simulations.
Key Insights and Ideas
- Semantic Strength vs. Numerical Weakness: Models are surprisingly good at "reading" a city—identifying landmarks or understanding human dynamics from images. However, they fail at "calculating" a city. Tasks requiring precise numerical abilities, like geospatial prediction or traffic control, saw models lagging behind traditional, non-LLM baselines.
- The Geospatial Bias: There is a clear "digital divide" in model performance. Models perform significantly better in well-documented Western cities (New York, London) than in cities with smaller online footprints (Nairobi, Cape Town). This suggests LLMs inherit the geographical biases present in their training data.
- VLM Performance Trade-offs: Interestingly, the process of fine-tuning an LLM to handle visual data (turning it into a VLM) often degrades its performance on purely textual urban tasks.
- No Single "Urban King": While GPT-4o is a strong all-rounder, it doesn't dominate every category. Open-source models like InternVL2 and Llama3-70B frequently outperformed proprietary models on specific tasks, highlighting the heterogeneity of urban challenges.
Key Results
- GeoQA: Even the best models achieved an accuracy of only ~0.4, indicating they lack detailed, conceptual knowledge of urban elements.
- Traffic Control: In the traffic signal control task, the top-performing VLM trailed the classic "Max-Pressure" algorithm by a staggering 41.9% in terms of junction queue length.
- Population Estimation: GPT-4o, the best-performing model for geospatial prediction, was still 18% less accurate (in terms of RMSE) than the specialized RemoteCLIP baseline.
- Error Analysis: The researchers identified frequent "logic errors" and "format errors." For example, models would sometimes suggest invalid traffic signal options or refuse to answer geolocalization queries due to "excessive alignment" (safety filters being too sensitive to location data).
Limitations and Future Work
The authors acknowledge that the quality of public data varies significantly across regions, which can skew results. Furthermore, the current benchmark is biased toward cities with high data availability.
Future work will focus on expanding the diversity of tasks and cities. Most importantly, the paper concludes that general-purpose LLMs are currently insufficient for high-stakes urban management. There is an urgent need to develop foundation models specifically tailored for the urban domain that can handle both the semantic nuances and the rigorous numerical demands of city life.
Summary updated