TUM2TWIN: Introducing the Large-Scale Multimodal Urban Digital Twin Benchmark Dataset

Olaf Wysocki, Benedikt Schwab, Manoj Kumar Biswanath, Michael Greza, Qilin Zhang, Jingwei Zhu, Thomas Froech, Medhini Heeramaglore, Ihab Hijazi, Khaoula Kanna, Mathias Pechinger, Zhaiyu Chen, Yao Sun, Alejandro Rueda Segura, Ziyang Xu, Omar AbdelGafar, Mansour Mehranfar, Chandan Yeshwanth, Yueh-Cheng Liu, Hadi Yazdi, Jiapan Wang, Stefan Auer, Katharina Anders, Klaus Bogenberger, André Borrmann, Angela Dai, Ludwig Hoegner, Christoph Holst, Thomas H. Kolbe, Ferdinand Ludwig, Matthias Nießner, Frank Petzold, Xiao Xiang Zhu, Boris Jutzi

TL;DR

TUM2TWIN is a massive, first-of-its-kind multimodal benchmark dataset designed to revolutionize how we build and validate Urban Digital Twins (UDTs). Spanning 100,000 $m^2$ of Munich, Germany, it provides 767 GB of georeferenced, semantically aligned data—ranging from satellite imagery to centimeter-accurate indoor laser scans—to bridge the gap between raw sensor observations and high-fidelity 3D city models.

The Challenge of Urban Digital Twins

Creating an Urban Digital Twin (UDT) is a significant engineering challenge. It requires the seamless integration of heterogeneous data from satellites, aircraft, drones, and ground vehicles, followed by the conversion of this raw data into semantically rich 3D models, such as CityGML.

Historically, research has been limited by "siloed" datasets. Existing benchmarks often provide high-quality point clouds without corresponding imagery, or 3D models lacking the original sensor data used for their creation. This fragmentation makes it difficult to validate end-to-end UDT workflows. TUM2TWIN addresses this by providing a comprehensive, co-registered pipeline that encompasses the entire lifecycle of a digital twin: acquisition, reconstruction, maintenance, and application.

Comparison of TUM2TWIN with existing urban datasets, demonstrating its unique combination of multitemporal, multimodal, and georeferenced semantic data.

Methodology: A Multi-Pillar Approach

The authors developed TUM2TWIN using a "multi-pillar" strategy for data acquisition and processing:

  • Multimodal Acquisition: Data was collected using a diverse fleet of sensors, including terrestrial laser scanners (TLS), mobile mapping systems (MLS) on vehicles and backpacks, uncrewed aerial systems (UAS), and Sentinel-1/2 satellites.
  • Georeferencing as the Core: Every data point is anchored to a global coordinate system (WGS 84 / UTM). This global "glue" allows for the fusion of disparate data types, such as aligning thermal imagery from a ground vehicle with a 3D mesh generated from aerial drone footage.
  • Semantic Modeling: Raw data was transformed into standardized 3D city models across multiple Levels of Detail (LoD1, LoD2, and LoD3) using the CityGML 3.0 standard, ensuring interoperability with urban planning software.
  • Simulation Integration: To address the "sim-to-real" gap, the dataset includes simulated Aerial Laser Scanning (ALS) data and synthetic satellite imagery.
The TUM2TWIN framework integrates point clouds, images, 3D models, and networks into a unified, georeferenced dataset.

Dataset Composition

TUM2TWIN consists of 32 distinct data subsets, totaling 767 GB. Key components include:

  • Point Clouds: High-precision TLS (TUM-TLS-24), multitemporal mobile scans (TUM-MLS-16/18/24), and UAS-based LiDAR.
  • Images: Street-level video, high-resolution facade images, thermal infrared sequences, and aerial orthophotos.
  • ZAHA Subset: A specialized dataset for facade segmentation containing 601 million annotated points across 15 semantic classes.
  • 3D Models: Georeferenced LoD1-3 building models, CAD models, and a high-density 3D mesh of the TUM campus.
  • Networks: High-definition (HD) maps of the road network designed for driving simulations.

Insights and Empirical Results

  • Multitemporal Analysis: By including data spanning from 2016 to 2024, TUM2TWIN enables research into change detection and the temporal evolution of digital twins.
  • The Necessity of Global Georeferencing: The authors demonstrate that global coordinates are essential for cross-platform data fusion (e.g., drone-to-satellite alignment).
  • LoD3 Precision: While most city models are limited to LoD2, TUM2TWIN provides LoD3 models with detailed architectural features, which are vital for solar potential analysis and flood modeling.
  • Novel View Synthesis: Combining high-accuracy TLS with UAS imagery significantly enhances the performance of techniques like Neural Radiance Fields (NeRF) and Gaussian Splatting in complex urban environments.
  • Accuracy: The dataset achieves centimeter-level global accuracy, with 7.1 mm absolute accuracy for TLS ground control points.

Limitations and Future Work

Currently, the dataset represents a "snapshot" in time and lacks real-time IoT feeds or live observation data (e.g., traffic cameras). Furthermore, while some indoor data is included, most building interiors are not yet modeled as full Building Information Modeling (BIM) objects. Future iterations of TUM2TWIN aim to incorporate "live" assets and deeper BIM integration to move toward a truly dynamic, "living" digital twin.

Summary updated

BibTeX
@inproceedings{wysocki2026tum,
  author = {Olaf Wysocki and Benedikt Schwab and Manoj Kumar Biswanath and Michael Greza and Qilin Zhang and Jingwei Zhu and Thomas Froech and Medhini Heeramaglore and Ihab Hijazi and Khaoula Kanna and Mathias Pechinger and Zhaiyu Chen and Yao Sun and Alejandro Rueda Segura and Ziyang Xu and Omar AbdelGafar and Mansour Mehranfar and Chandan Yeshwanth and Yueh-Cheng Liu and Hadi Yazdi and Jiapan Wang and Stefan Auer and Katharina Anders and Klaus Bogenberger and André Borrmann and Angela Dai and Ludwig Hoegner and Christoph Holst and Thomas H. Kolbe and Ferdinand Ludwig and Matthias Nießner and Frank Petzold and Xiao Xiang Zhu and Boris Jutzi},
  title = {{TUM2TWIN: Introducing the Large-Scale Multimodal Urban Digital Twin Benchmark Dataset}},
  booktitle = {ISPRS},
  year = {2026},
  doi = {10.1016/j.isprsjprs.2025.12.013},
}
Figure