CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks

Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, Yong Li

TL;DR

CityBench is the first systematic, interactive benchmark designed to evaluate Large Language Models (LLMs) and Vision-Language Models (VLMs) across a diverse range of urban tasks. By testing 30 models in 13 global cities, the researchers found that while current models excel at semantic understanding, they struggle significantly with domain-specific numerical reasoning and complex urban decision-making.

What Problem Does This Paper Address?

As we rush to integrate LLMs into everything from urban planning to autonomous traffic management, a critical question remains: how capable are these models in the messy, dynamic, and multi-modal reality of a city?

Existing evaluations for LLMs in the urban domain are often fragmented—focusing on a single city, a single data type (like just text or just satellite images), or static tasks that don't account for the interactive nature of urban environments. The challenge lies in the sheer diversity of urban data and the highly dynamic nature of city life. This paper introduces CityBench to bridge this gap, providing a scalable and systematic platform to see if "world models" actually understand the urban world.

Methods

The authors developed a three-tiered framework to move beyond static question-answering:

  1. CityData: A massive multi-source data collector that integrates geospatial data (OpenStreetMap), visual data (Google/Baidu street views and Esri satellite imagery), and human activity data (Foursquare check-ins and synthetic movement patterns).
  2. CitySimu: An interactive simulation engine. Unlike static benchmarks, CitySimu allows models to act as agents. It simulates individual mobility, visual environments, and microscopic traffic dynamics, providing APIs for models to "sense" the environment and "act" (e.g., changing a traffic signal).
  3. CityBench: The evaluation suite itself, comprising 8 representative tasks split into two categories: Perception & Understanding (like geolocalization and infrastructure inference) and Planning & Decision-Making (like traffic signal control and urban exploration).
The CityBench framework, integrating data collection, simulation, and multi-modal evaluation.

Datasets

The benchmark utilizes CityData, which covers 13 major global cities including New York, London, Tokyo, Nairobi, and Shanghai. The scale is impressive:

  • 20K+ images (satellite and street view).
  • 30K+ questions generated through a mix of templates and LLM-assisted methods.
  • Human activity data from Foursquare and synthetic OD (Origin-Destination) matrices to power realistic movement simulations.

Key Insights and Ideas

  • Semantic Strength vs. Numerical Weakness: Models are surprisingly good at "reading" a city—identifying landmarks or understanding human dynamics from images. However, they fail at "calculating" a city. Tasks requiring precise numerical abilities, like geospatial prediction or traffic control, saw models lagging behind traditional, non-LLM baselines.
  • The Geospatial Bias: There is a clear "digital divide" in model performance. Models perform significantly better in well-documented Western cities (New York, London) than in cities with smaller online footprints (Nairobi, Cape Town). This suggests LLMs inherit the geographical biases present in their training data.
  • VLM Performance Trade-offs: Interestingly, the process of fine-tuning an LLM to handle visual data (turning it into a VLM) often degrades its performance on purely textual urban tasks.
  • No Single "Urban King": While GPT-4o is a strong all-rounder, it doesn't dominate every category. Open-source models like InternVL2 and Llama3-70B frequently outperformed proprietary models on specific tasks, highlighting the heterogeneity of urban challenges.
Detailed performance results showing significant variations across different cities, highlighting geospatial bias.

Key Results

  • GeoQA: Even the best models achieved an accuracy of only ~0.4, indicating they lack detailed, conceptual knowledge of urban elements.
  • Traffic Control: In the traffic signal control task, the top-performing VLM trailed the classic "Max-Pressure" algorithm by a staggering 41.9% in terms of junction queue length.
  • Population Estimation: GPT-4o, the best-performing model for geospatial prediction, was still 18% less accurate (in terms of RMSE) than the specialized RemoteCLIP baseline.
  • Error Analysis: The researchers identified frequent "logic errors" and "format errors." For example, models would sometimes suggest invalid traffic signal options or refuse to answer geolocalization queries due to "excessive alignment" (safety filters being too sensitive to location data).
Common errors found in model responses, including logic errors in mobility prediction and format errors in traffic control.

Limitations and Future Work

The authors acknowledge that the quality of public data varies significantly across regions, which can skew results. Furthermore, the current benchmark is biased toward cities with high data availability.

Future work will focus on expanding the diversity of tasks and cities. Most importantly, the paper concludes that general-purpose LLMs are currently insufficient for high-stakes urban management. There is an urgent need to develop foundation models specifically tailored for the urban domain that can handle both the semantic nuances and the rigorous numerical demands of city life.

Summary updated

Abstract

As large language models (LLMs) continue to advance and gain widespread use, establishing systematic and reliable evaluation methodologies for LLMs and vision-language models (VLMs) has become essential to ensure their real-world effectiveness and reliability. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design \textit{CityBench}, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build \textit{CityData} to integrate the diverse urban data and \textit{CitySimu} to simulate fine-grained urban dynamics. Based on \textit{CityData} and \textit{CitySimu}, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the \textit{CityBench}. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level numerical abilities, e.g., geospatial prediction and traffic control task.
BibTeX
@inproceedings{feng2025citybench,
  author = {Jie Feng and Jun Zhang and Tianhui Liu and Xin Zhang and Tianjian Ouyang and Junbo Yan and Yuwei Du and Siqi Guo and Yong Li},
  title = {{CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
  doi = {10.1145/3711896.3737375},
}
Figure