Mobile Robot Oriented Large-Scale Indoor Dataset for Dynamic Scene Understanding

Yifan Tang, Cong Tai, Fang-xing Chen, Wanting Zhang, Tao Zhang, Xue-Ping Liu, Yongjin Liu, Long Zeng

TL;DR

The THUD (Tsinghua University Dynamic) robotic dataset bridges a critical gap in mobile robotics by providing a large-scale indoor RGB-D dataset specifically designed for dynamic environments. Featuring over 90K frames across 13 real and synthetic scenarios, it includes 20 million 2D/3D bounding boxes to help robots navigate crowded spaces like cafeterias and supermarkets. Evaluation on mainstream algorithms reveals significant performance drops when facing moving obstacles, highlighting the urgent need for robust dynamic scene understanding.

What Problem Does This Paper Address?

Most existing indoor robotic datasets (such as ScanNet, NYU-Depth v2, and SUN RGB-D) focus almost exclusively on static scenes. However, the real-world environments where service robots are deployed—like supermarkets, restaurants, and campus buildings—are highly dynamic, filled with moving pedestrians, shopping carts, and other robots.

When algorithms trained on static data are introduced to these crowded spaces, they frequently degrade or fail entirely. Tasks like robot relocalization and 3D object detection become highly unstable due to the visual and geometric noise introduced by moving objects. To build robots that can safely operate around humans, the research community needs high-quality, large-scale benchmarks that combine multi-modal data (RGB-D, IMU, poses) with dense annotations of both static and dynamic objects.

Methods

The authors developed a comprehensive data acquisition and annotation pipeline spanning both physical and virtual worlds:

  • Real-World Data Acquisition: The team built a mobile data collection platform using a PuduBot2 service robot integrated with a Microsoft Kinect V2 camera and a Jetson AGX processing unit. The PuduBot2 provided real-time wheel odometry and v-SLAM localization, while the Kinect captured RGB-D image streams.
  • Synthetic Data Acquisition: To safely simulate hazardous or highly crowded environments, they built a physical simulation platform in Unity3D. The virtual robot mirrored the exact sensor height, pitch, and resolution of the real-world platform, publishing synthetic data and IMU readings via ROS2.
  • Annotation Pipeline: Synthetic data annotations were automatically generated by the simulation engine, ensuring perfect ground-truth accuracy. For real-world data, the authors used a semi-automated approach: initial 2D/3D bounding boxes were predicted using models like Faster R-CNN and VoteNet, followed by meticulous manual review and pixel-wise correction by human annotators.
Fig. 1: THUD robotic dataset, first column: real and synthetic data acquisition platforms; second column: scene classes and annotation types; third column: applications supported by THUD.

Datasets

The paper introduces the THUD Robotic Dataset, which stands out from prior benchmarks due to its explicit focus on dynamic complexity:

  • Scale: 90,175 annotated frames (84,984 synthetic, 5,191 real-world).
  • Scenarios: 13 large-scale indoor environments (8 real-world campus scenes like canteens, laboratories, and cafes; 5 synthetic scenes like supermarkets and offices) with an average area exceeding 300 $m^2$.
  • Annotations: Over 20 million total labels, including 1.2 million labels specifically tracking dynamic objects (pedestrians, robots, shopping carts) across 91 object categories.

Key Insights and Ideas

  • The Dynamic Performance Gap: Standard 3D object detection algorithms show a massive performance drop when switching from static to dynamic objects. For example, the DeMF algorithm drops from 45.43 mAP on static objects down to 28.56 mAP on dynamic objects in the canteen scene.
  • Density Compounds the Challenge: The authors introduced a "dynamic complexity" metric (average number of dynamic pedestrians per frame). As this complexity increases, the performance gap between static and dynamic object detection widens significantly.
  • Relocalization Vulnerability: Local-feature-based relocalization methods (like FeatLoc) are incredibly sensitive to dynamic noise compared to global methods (like NetVLAD). Moving people introduce visual disturbances that corrupt the robot's trajectory tracking, causing median translation and rotation errors to spike.
  • Segmentation Resilience: Interestingly, semantic segmentation algorithms (like ESANet) proved relatively resilient to dynamic objects, showing only minor drops in Mean Intersection over Union (MIoU). This indicates that pixel-wise classification is less disrupted by motion than geometric localization or bounding box estimation.

Key Results

The authors benchmarked mainstream algorithms across three core tasks on the THUD dataset:

3D Object Detection

Evaluated on F-PointNet, ImVoteNet, and DeMF. In the real-world canteen scene, DeMF achieved the highest performance but still demonstrated a stark contrast: 45.43 mAP for static objects versus 28.56 mAP for dynamic objects.

Semantic Segmentation

Tested using ACNet, RedNet, ESANet, and SA-Gate. While SA-Gate performed exceptionally well on the synthetic supermarket scene (83.19% MIoU), its performance dropped sharply on the real-world canteen scene (58.34% MIoU), highlighting a significant reality gap.

Robot Relocalization

Tested using NetVLAD and FeatLoc. In crowded real-world scenes, tracking errors spiked dramatically. FeatLoc, for instance, suffered a median translation error of 2.41 meters and a rotation error of 32.4° in the canteen environment due to pedestrian interference.

Limitations and Future Work

While THUD represents a major step forward, the authors note a few limitations:

  1. Real-World Volume: The real-world portion of the dataset (5,191 frames) is still relatively small compared to the synthetic data (nearly 85K frames).
  2. Hardware Instability: Due to the inherent limitations of the Kinect V2 sensor, the real-world frame rate fluctuated between 15–30 fps, which can introduce temporal jitter.

Future Work: The team plans to continuously expand the dataset to include more crowded real-world scenarios. They also aim to extend the benchmark to support downstream dynamic tasks such as multi-object tracking, trajectory prediction, and online robotic navigation.

Summary updated

Abstract

Most existing robotic datasets capture static scene data and thus are limited in evaluating robots' dynamic performance. To address this, we present a mobile robot oriented large-scale indoor dataset, denoted as THUD (Tsinghua University Dynamic) robotic dataset, for training and evaluating their dynamic scene understanding algorithms. Specifically, the THUD dataset construction is first detailed, including organization, acquisition, and annotation methods. It comprises both real-world and synthetic data, collected with a real robot platform and a physical simulation platform, respectively. Our current dataset includes 13 larges-scale dynamic scenarios, 90K image frames, 20M 2D/3D bounding boxes of static and dynamic objects, camera poses, and IMU. The dataset is still continuously expanding. Then, the performance of mainstream indoor scene understanding tasks, e.g. 3D object detection, semantic segmentation, and robot relocalization, is evaluated on our THUD dataset. These experiments reveal serious challenges for some robot scene understanding tasks in dynamic scenes. By sharing this dataset, we aim to foster and iterate new mobile robot algorithms quickly for robot actual working dynamic environment, i.e. complex crowded dynamic scenes.
BibTeX
@inproceedings{tang2024mobile,
  author = {Yifan Tang and Cong Tai and Fang-xing Chen and Wanting Zhang and Tao Zhang and Xue-Ping Liu and Yongjin Liu and Long Zeng},
  title = {{Mobile Robot Oriented Large-Scale Indoor Dataset for Dynamic Scene Understanding}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611489},
}
Figure