Abstract
We present UrbanScene3D, a large-scale data platform for research of urban scene perception and reconstruction. UrbanScene3D contains over 128k high-resolution images covering 16 scenes including large-scale real urban regions and synthetic cities with 136 km^2 area in total. The dataset also contains high-precision LiDAR scans and hundreds of image sets with different observation patterns, which provide a comprehensive benchmark to design and evaluate aerial path planning and 3D reconstruction algorithms. In addition, the dataset, which is built on Unreal Engine and Airsim simulator together with the manually annotated unique instance label for each building in the dataset, enables the generation of all kinds of data, e.g., 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations, etc. The simulator with physical engine and lighting system not only produce variety of data but also enable users to simulate cars or drones in the proposed urban environment for future research.
Introduction
While 3D computer vision has flourished in indoor environments thanks to datasets like ScanNet and S3DIS, outdoor urban scenes remain a "wild west." Capturing high-quality outdoor data is notoriously difficult due to the sheer scale of cities, unpredictable lighting, and the limitations of ground-based sensors (like those on self-driving cars), which often miss the upper structures of buildings.
Existing drone-based datasets typically focus on narrow tasks like object detection or tracking rather than full-scale 3D reconstruction. Furthermore, synthetic datasets often lack the textural and geometric "messiness" of the real world, leading to a significant "sim-to-real" gap where algorithms that work in a simulator fail on the street. UrbanScene3D tackles these challenges by providing a unified platform for capturing, reconstructing, and simulating complex urban environments.
The UrbanScene3D Dataset
The primary contribution is the UrbanScene3D dataset, which covers 16 scenes across a total area of 136 $km^2$. It includes over 128,000 high-resolution images, high-precision LiDAR scans, and manually annotated building instance labels. Unlike ground-based datasets like KITTI or CAD-based datasets like HoliCity, UrbanScene3D offers complete 360-degree aerial coverage and high-fidelity textures.
Methodology
The authors took a two-pronged approach to data generation:
- Synthetic Modeling: Professional artists created 10 massive, compact synthetic city models (e.g., "New York," "Chicago") within Unreal Engine.
- Real-World Capture: Professional-grade drones (DJI M300RTK and Phantom 4 RTK) were used to capture 6 real-world scenes. To provide "ground truth" for these real scenes, a high-precision Trimble X7 LiDAR scanner created millimeter-accurate point clouds.
- Path Planning Evaluation: A core component of the paper is evaluating how different drone flight patterns affect 3D reconstruction. They tested four distinct planners—ranging from standard "S-shaped" oblique photography to complex, optimization-based algorithms.
- Simulation Integration: The entire dataset is integrated with the AirSim simulator, allowing researchers to simulate physics (gravity, collisions) and environmental factors (weather, time of day) for autonomous drone or car navigation.
Key Insights and Results
- The "Proxy" Sensitivity: Most advanced aerial path planners require a "proxy" (a coarse 3D model) to plan a flight. The authors found that reconstruction quality improves significantly as the proxy becomes more detailed.
- Efficiency vs. Quality Trade-off: Standard "Zig-Zag" flight patterns are the most energy-efficient for drones but result in the "roughest" reconstructions because they cannot "see" into narrow spaces between buildings.
- Sim-to-Real Benchmark: By providing both perfect CAD models (synthetic) and high-precision LiDAR scans (real), the dataset allows researchers to measure exactly how much performance drops when moving from clean simulated data to noisy real-world data.
- Accuracy: Real-world LiDAR registration achieved an overall error of just 3mm to 6mm, providing a near-perfect baseline for evaluating reconstruction algorithms.
- Scale: The dataset includes three large-scale city scenes covering over 24 $km^2$ each, far exceeding the scale of previous benchmarks.
Conclusion and Future Work
While UrbanScene3D is a major step forward, the authors note that the current version focuses primarily on raw geometry and textures. Future directions include adding more "semantic" geometric descriptions, such as 3D structural points and wire-frames. Additionally, while the simulator supports moving cars and drones, the static dataset primarily focuses on the "built" environment. Integrating more dynamic agents (pedestrians, traffic) into the benchmarks would further benefit autonomous driving research.
Summary updated