UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator

Yilin Liu, Fuyou Xue, Hui Huang

Abstract

We present UrbanScene3D, a large-scale data platform for research of urban scene perception and reconstruction. UrbanScene3D contains over 128k high-resolution images covering 16 scenes including large-scale real urban regions and synthetic cities with 136 km² area in total. The dataset also contains high-precision LiDAR scans and hundreds of image sets with different observation patterns, which provide a comprehensive benchmark to design and evaluate aerial path planning and 3D reconstruction algorithms. In addition, the dataset, which is built on Unreal Engine and Airsim simulator together with the manually annotated unique instance label for each building in the dataset, enables the generation of all kinds of data, e.g., 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations, etc. The simulator with physical engine and lighting system not only produce variety of data but also enable users to simulate cars or drones in the proposed urban environment for future research.

Introduction

While indoor 3D computer vision has flourished thanks to datasets like ScanNet and S3DIS, outdoor urban reconstruction remains a significant challenge. Capturing cities is difficult due to their immense scale, unpredictable weather, and the limitations of ground-based sensors, which often have restricted fields of view. Drones (UAVs) offer a solution, but most existing UAV datasets focus on object detection or tracking rather than full-scene reconstruction. Furthermore, there is a lack of standardized benchmarks to determine the optimal way to fly a drone to capture a building. UrbanScene3D fills this gap by providing a unified platform for planning, capturing, and simulating urban environments.

The UrbanScene3D Ecosystem

The authors built a multi-faceted ecosystem consisting of:

  • The Dataset: 16 diverse scenes (10 synthetic CAD-based cities and 6 real-world reconstructions).
  • The Simulator: Built on Unreal Engine and AirSim, allowing researchers to simulate drone or car physics, manipulate weather/lighting, and generate "perfect" labels (depth maps, 2D/3D bounding boxes) for training.
  • Path Planning Benchmark: Evaluation of four different aerial path planning algorithms (ranging from standard "Zig-Zag" oblique photography to complex optimization methods).
  • Proxy Evaluation: Testing how the accuracy of "proxies"—rough 3D models used to plan a flight—impacts the final reconstruction quality.

Dataset Composition

The UrbanScene3D dataset includes over 128k high-resolution images covering 136 km². It is divided into synthetic and real-world components.

Real-World Scenes

The real-world component consists of high-fidelity reconstructions of locations like "Hospital," "Polytech," and "ArtSci" using professional-grade drones (DJI M300RTK). For these scenes, the researchers used a Trimble X7 laser scanner to create millimeter-accurate LiDAR point clouds, serving as the ground truth for evaluation.

Overview of diverse real-world scenes included in the UrbanScene3D dataset.

Synthetic Scenes

The synthetic component features models of major cities like New York, Chicago, and San Francisco. These CAD-based models provide perfect ground truth geometry, allowing for the generation of various labels such as 2D/3D bounding boxes and semantic segmentations.

Synthetic city models and specific architectural assets used for large-scale simulation.

Key Insights

  • The "Proxy" is Paramount: The quality of the initial rough map used to plan a drone's flight significantly dictates the final result. As proxies move from simple "box" models to fine meshes, the resulting 3D reconstructions show marked improvements in completeness and accuracy.
  • Closing the Sim-to-Real Gap: Unlike most simulators that only use virtual worlds, UrbanScene3D allows users to import real-world reconstructed meshes back into the simulator. This lets researchers train autonomous agents in digital twins of actual locations.
  • Efficiency vs. Quality: Industry-standard "Oblique Photography" (S-shaped paths) is energy-efficient but fails in complex urban "canyons." Specialized planners offer a better balance of energy consumption and reconstruction detail.

Results and Future Work

The dataset includes three large-scale city scenes covering over 24 km² each, far exceeding previous benchmarks. Real-world registration errors for the LiDAR ground truth were kept as low as 3mm to 6mm. The authors demonstrated that optimized path planners can achieve higher reconstruction accuracy and completeness than manual or grid-based patterns.

Future work will focus on adding high-level geometric descriptions, such as 3D structural points and wireframes, and moving toward holistic scene understanding where AI recognizes structural components during reconstruction.

Summary updated

Abstract

We present UrbanScene3D, a large-scale data platform for research of urban scene perception and reconstruction. UrbanScene3D contains over 128k high-resolution images covering 16 scenes including large-scale real urban regions and synthetic cities with 136 km^2 area in total. The dataset also contains high-precision LiDAR scans and hundreds of image sets with different observation patterns, which provide a comprehensive benchmark to design and evaluate aerial path planning and 3D reconstruction algorithms. In addition, the dataset, which is built on Unreal Engine and Airsim simulator together with the manually annotated unique instance label for each building in the dataset, enables the generation of all kinds of data, e.g., 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations, etc. The simulator with physical engine and lighting system not only produce variety of data but also enable users to simulate cars or drones in the proposed urban environment for future research.
BibTeX
@inproceedings{liu2022urbanscene,
  author = {Yilin Liu and Fuyou Xue and Hui Huang},
  title = {{UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20074-8\_6},
}
Figure