Abstract
Standard indoor computer vision datasets often focus on static environments, failing to account for the dynamic complexity encountered by mobile service robots. We present THUD, a large-scale indoor dataset from Tsinghua University designed to bridge this gap. THUD comprises over 90,000 frames of real-world and synthetic data with more than 20 million 2D/3D annotations. By explicitly modeling dynamic elements like pedestrians and moving objects, THUD provides a rigorous benchmark for tasks such as 3D object detection, semantic segmentation, and robot relocalization in crowded environments.
Introduction
Most existing indoor datasets capture "frozen" scenes where the camera is the only moving entity. However, real-world service robots operate in chaotic environments—cafeterias, supermarkets, and laboratories—where people move, furniture is rearranged, and other robots interact. This "dynamic complexity" significantly degrades the performance of algorithms trained on static data. When these algorithms encounter moving obstacles, their ability to detect objects or determine their own location often plummets.
The THUD (Tsinghua University Dynamic) dataset addresses this by providing a diverse set of scenarios with varying levels of pedestrian density and dynamic obstacles. It aims to prepare robots for the messy reality of human-centric spaces by bridging the gap between sterile lab datasets and the moving real world.
Methodology
The researchers employed a hybrid data acquisition strategy to balance real-world fidelity with the scalability and safety of simulation.
Real-World Acquisition
A PUDUbot2 service robot served as the primary physical platform. It was equipped with a Kinect V2 sensor, a Jetson AGX for processing, and a high-capacity battery. The robot navigated through eight campus locations, including canteens and labs, at different times of day to capture a wide spectrum of pedestrian activity.
Synthetic Acquisition
Using the Unity3D engine, the team developed five large-scale virtual environments, such as supermarkets and offices. Simulation allows for the safe capture of "edge cases"—such as children running or navigation near glass doors and stairs—that are difficult or hazardous to record in the real world.
Annotation Pipeline
For synthetic data, labels (2D/3D boxes, semantic/instance masks) were automatically generated by the engine, ensuring perfect ground truth. For real-world data, a semi-automated pipeline was used: initial labels from state-of-the-art models like Faster R-CNN were meticulously corrected by human annotators to ensure high precision across 91 object categories.
The THUD Dataset
THUD is characterized by its scale and diversity, offering a more realistic challenge than predecessors like ScanNet or NYU-Depth v2:
- Volume: 90,175 total frames (84,984 synthetic, 5,191 real-world).
- Annotations: Over 20 million labels across 91 object categories.
- Scenarios: 13 distinct environments ranging from quiet offices to bustling supermarkets.
- Dynamic Metrics: Unlike static datasets, THUD includes a metric for "dynamic complexity" based on pedestrian density, allowing researchers to quantify how movement affects model performance.
Key Insights and Results
Benchmarking mainstream algorithms on THUD revealed several critical challenges for modern robotics:
- The "Dynamic Tax": Models consistently struggle with moving objects. In canteen tests, the DeMF model's precision dropped from 45.43 mAP for static objects to 28.56 mAP for dynamic ones.
- Relocalization Fragility: Robot self-localization accuracy (using models like NetVlad and FeatLoc) degrades linearly as pedestrian density increases. Moving features introduce "noise" that disrupts the robot's ability to orient itself.
- Sim-to-Real Gap: While synthetic data provides a robust stress test for hazards like glass walls, a performance gap remains. For instance, semantic segmentation mIoU dropped from 83.19% in simulation to 58.34% in real-world scenes, highlighting the difficulty of real-world lighting and occlusions.
Conclusion
The THUD dataset provides a vital resource for the robotics community, moving beyond static scene understanding toward true dynamic awareness. By providing high-quality annotations in complex, crowded environments, it enables the development of more robust navigation and perception systems for the next generation of service robots. Future work will focus on expanding the dataset to include trajectory prediction and more complex autonomous navigation tasks in crowded spaces.
Summary updated