Mobile Robot Oriented Large-Scale Indoor Dataset for Dynamic Scene Understanding

Yifan Tang, Cong Tai, Fang-xing Chen, Wanting Zhang, Tao Zhang, Xue-Ping Liu, Yong-Jin Liu, Long Zeng

Abstract

Standard indoor computer vision datasets often focus on static environments, failing to account for the dynamic complexity encountered by mobile service robots. We present THUD, a large-scale indoor dataset from Tsinghua University designed to bridge this gap. THUD comprises over 90,000 frames of real-world and synthetic data with more than 20 million 2D/3D annotations. By explicitly modeling dynamic elements like pedestrians and moving objects, THUD provides a rigorous benchmark for tasks such as 3D object detection, semantic segmentation, and robot relocalization in crowded environments.

Introduction

Most existing indoor datasets capture "frozen" scenes where the camera is the only moving entity. However, real-world service robots operate in chaotic environments—cafeterias, supermarkets, and laboratories—where people move, furniture is rearranged, and other robots interact. This "dynamic complexity" significantly degrades the performance of algorithms trained on static data. When these algorithms encounter moving obstacles, their ability to detect objects or determine their own location often plummets.

The THUD (Tsinghua University Dynamic) dataset addresses this by providing a diverse set of scenarios with varying levels of pedestrian density and dynamic obstacles. It aims to prepare robots for the messy reality of human-centric spaces by bridging the gap between sterile lab datasets and the moving real world.

Overview of the THUD dataset pipeline, including acquisition platforms, scene diversity, ground truth annotations, and supported robotic applications.

Methodology

The researchers employed a hybrid data acquisition strategy to balance real-world fidelity with the scalability and safety of simulation.

Real-World Acquisition

A PUDUbot2 service robot served as the primary physical platform. It was equipped with a Kinect V2 sensor, a Jetson AGX for processing, and a high-capacity battery. The robot navigated through eight campus locations, including canteens and labs, at different times of day to capture a wide spectrum of pedestrian activity.

The real-world data acquisition platform based on the PUDUbot2, featuring a Kinect v2 sensor and Jetson AGX.

Synthetic Acquisition

Using the Unity3D engine, the team developed five large-scale virtual environments, such as supermarkets and offices. Simulation allows for the safe capture of "edge cases"—such as children running or navigation near glass doors and stairs—that are difficult or hazardous to record in the real world.

Annotation Pipeline

For synthetic data, labels (2D/3D boxes, semantic/instance masks) were automatically generated by the engine, ensuring perfect ground truth. For real-world data, a semi-automated pipeline was used: initial labels from state-of-the-art models like Faster R-CNN were meticulously corrected by human annotators to ensure high precision across 91 object categories.

The THUD Dataset

THUD is characterized by its scale and diversity, offering a more realistic challenge than predecessors like ScanNet or NYU-Depth v2:

  • Volume: 90,175 total frames (84,984 synthetic, 5,191 real-world).
  • Annotations: Over 20 million labels across 91 object categories.
  • Scenarios: 13 distinct environments ranging from quiet offices to bustling supermarkets.
  • Dynamic Metrics: Unlike static datasets, THUD includes a metric for "dynamic complexity" based on pedestrian density, allowing researchers to quantify how movement affects model performance.

Key Insights and Results

Benchmarking mainstream algorithms on THUD revealed several critical challenges for modern robotics:

  1. The "Dynamic Tax": Models consistently struggle with moving objects. In canteen tests, the DeMF model's precision dropped from 45.43 mAP for static objects to 28.56 mAP for dynamic ones.
  2. Relocalization Fragility: Robot self-localization accuracy (using models like NetVlad and FeatLoc) degrades linearly as pedestrian density increases. Moving features introduce "noise" that disrupts the robot's ability to orient itself.
  3. Sim-to-Real Gap: While synthetic data provides a robust stress test for hazards like glass walls, a performance gap remains. For instance, semantic segmentation mIoU dropped from 83.19% in simulation to 58.34% in real-world scenes, highlighting the difficulty of real-world lighting and occlusions.

Conclusion

The THUD dataset provides a vital resource for the robotics community, moving beyond static scene understanding toward true dynamic awareness. By providing high-quality annotations in complex, crowded environments, it enables the development of more robust navigation and perception systems for the next generation of service robots. Future work will focus on expanding the dataset to include trajectory prediction and more complex autonomous navigation tasks in crowded spaces.

Summary updated

Abstract

Most existing robotic datasets capture static scene data and thus are limited in evaluating robots’ dynamic performance. To address this, we present a mobile robot oriented large-scale indoor dataset, denoted as THUD (Tsinghua University Dynamic) robotic dataset, for training and evaluating their dynamic scene understanding algorithms. Specifically, the THUD dataset construction is first detailed, including organization, acquisition, and annotation methods. It comprises both real-world and synthetic data, collected with a real robot platform and a physical simulation platform, respectively. Our current dataset includes 13 larges-scale dynamic scenarios, 90K image frames, 20M 2D/3D bounding boxes of static and dynamic objects, camera poses, and IMU. The dataset is still continuously expanding. Then, the performance of mainstream indoor scene understanding tasks, e.g. 3D object detection, semantic segmentation, and robot relocalization, is evaluated on our THUD dataset. These experiments reveal serious challenges for some robot scene understanding tasks in dynamic scenes. By sharing this dataset, we aim to foster and iterate new mobile robot algorithms quickly for robot actual working dynamic environment, i.e. complex crowded dynamic scenes.
BibTeX
@inproceedings{tang2024mobile,
  author = {Yifan Tang and Cong Tai and Fang-xing Chen and Wanting Zhang and Tao Zhang and Xue-Ping Liu and Yong-Jin Liu and Long Zeng},
  title = {{Mobile Robot Oriented Large-Scale Indoor Dataset for Dynamic Scene Understanding}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611489},
}
Figure