Abstract
Large language models (LLMs) with powerful generalization ability have been widely used in many domains. A systematic and reliable evaluation of LLMs is a crucial step in their development and applications, especially for specific professional fields. In the urban domain, there have been some early explorations about the usability of LLMs, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for the urban domain lies in the diversity of data and scenarios, as well as the complex and dynamic nature of cities. In this paper, we propose CityBench, an interactive simulator based evaluation platform, as the first systematic evaluation benchmark for the capability of LLMs for urban domain. First, we build CitySim to integrate the multi-source data and simulate fine-grained urban dynamics. Based on CitySim, we design 8 tasks in 2 categories of perception-understanding and decision-making group to evaluate the capability of LLMs as city-scale world model for urban domain. Due to the flexibility and ease-of-use of CitySim, our evaluation platform CityBench can be easily extended to any city in the world. We evaluate 13 well-known LLMs including open source LLMs and commercial LLMs in 13 cities around the world. Extensive experiments demonstrate the scalability and effectiveness of proposed CityBench and shed lights for the future development of LLMs in urban domain.
Introduction
As large language models (LLMs) and vision-language models (VLMs) move beyond simple chat interfaces and into "agentic" roles, researchers are increasingly asking: can these models actually understand and navigate the physical world? While we have benchmarks for coding, math, and general reasoning, the complex, dynamic environment of a city remains a frontier.
CityBench is the first systematic benchmark designed to evaluate how well LLMs and VLMs handle the unique challenges of urban tasks. It moves beyond static knowledge (e.g., "What are the coordinates of Paris?") to test if an LLM can actually act as a "world model"—predicting human mobility, controlling traffic lights, or navigating streets based on visual cues.
The CityBench Framework
Urban environments are a "final boss" for AI evaluation. They are multi-modal (combining maps, images, and human activity data), highly dynamic, and require a mix of high-level reasoning and precise numerical calculation. The researchers built a three-part framework to bridge this gap:
- CityData: A data integration module that pulls from OpenStreetMap (geospatial), Google/Baidu Maps (visual), and Foursquare/WorldPop (human activity) across 13 global cities.
- CitySimu: An interactive simulator that uses this data to create fine-grained urban dynamics. It includes sub-simulators for individual mobility, urban visual environments (street-view navigation), and microscopic traffic simulation.
- CityBench: The evaluation layer, featuring 8 tasks split into Perception & Understanding and Planning & Decision-making.
Tasks and Evaluation
The benchmark utilizes CityData, which covers 13 diverse cities worldwide, including Beijing, Tokyo, London, Nairobi, and New York. This global scope is critical for identifying regional biases in model performance. The dataset includes over 20,000 images and 30,000 evaluation questions.
Tasks are categorized into Perception & Understanding (e.g., image geolocalization, population density prediction) and Planning & Decision-making (e.g., traffic signal control, visual navigation). These tasks require models to process complex visual and textual instructions to make real-world decisions.
Key Insights and Results
The researchers evaluated 30 different models, including GPT-4o, Claude, and various open-source models like Llama-3 and InternVL2. Several critical findings emerged:
- The "Numerical Gap": LLMs are surprisingly good at semantic tasks (like identifying a city from a street view) but fail at tasks requiring high-level numerical abilities. For example, they struggle to predict population density from satellite images or optimize traffic flow as well as traditional, specialized algorithms.
- Geospatial Bias: There is a massive performance gap depending on the city. Models perform significantly better in "data-rich" cities like New York and London compared to Nairobi or Cape Town. This suggests models are "memorizing" rather than "reasoning" about urban space.
- VLM Performance Degradation: Interestingly, the authors found that fine-tuning models for vision often degrades their original textual reasoning performance on urban tasks.
- Top Performer: GPT-4o emerged as the strongest generalist, particularly in image geolocalization (86.2% city-level accuracy).
- The Baseline Deficit: In specialized tasks like traffic signal control, even the best LLMs trail behind classic baselines. The top-performing VLM trailed the "Max-Pressure" traffic control algorithm by 41.9% in terms of reducing queue lengths.
Conclusion
CityBench demonstrates that while current LLMs possess significant "urban commonsense," they are not yet "urban-native." They struggle with professional geospatial knowledge and precise numerical reasoning. The authors argue for the development of foundation models specifically tailored for the urban domain and highlight that better instruction-following is a prerequisite for practical urban deployment. Future work will focus on expanding task variety and incorporating more diverse global ground truth data.
Summary updated