TL;DR
The THUD (Tsinghua University Dynamic) robotic dataset bridges a critical gap in mobile robotics by providing a large-scale indoor RGB-D dataset specifically designed for dynamic environments. Featuring over 90K frames across 13 real and synthetic scenarios, it includes 20 million 2D/3D bounding boxes to help robots navigate crowded spaces like cafeterias and supermarkets. Evaluation on mainstream algorithms reveals significant performance drops when facing moving obstacles, highlighting the urgent need for robust dynamic scene understanding.
What Problem Does This Paper Address?
Most existing indoor robotic datasets (such as ScanNet, NYU-Depth v2, and SUN RGB-D) focus almost exclusively on static scenes. However, the real-world environments where service robots are deployed—like supermarkets, restaurants, and campus buildings—are highly dynamic, filled with moving pedestrians, shopping carts, and other robots.
When algorithms trained on static data are introduced to these crowded spaces, they frequently degrade or fail entirely. Tasks like robot relocalization and 3D object detection become highly unstable due to the visual and geometric noise introduced by moving objects. To build robots that can safely operate around humans, the research community needs high-quality, large-scale benchmarks that combine multi-modal data (RGB-D, IMU, poses) with dense annotations of both static and dynamic objects.
Methods
The authors developed a comprehensive data acquisition and annotation pipeline spanning both physical and virtual worlds:
- Real-World Data Acquisition: The team built a mobile data collection platform using a PuduBot2 service robot integrated with a Microsoft Kinect V2 camera and a Jetson AGX processing unit. The PuduBot2 provided real-time wheel odometry and v-SLAM localization, while the Kinect captured RGB-D image streams.
- Synthetic Data Acquisition: To safely simulate hazardous or highly crowded environments, they built a physical simulation platform in Unity3D. The virtual robot mirrored the exact sensor height, pitch, and resolution of the real-world platform, publishing synthetic data and IMU readings via ROS2.
- Annotation Pipeline: Synthetic data annotations were automatically generated by the simulation engine, ensuring perfect ground-truth accuracy. For real-world data, the authors used a semi-automated approach: initial 2D/3D bounding boxes were predicted using models like Faster R-CNN and VoteNet, followed by meticulous manual review and pixel-wise correction by human annotators.
Datasets
The paper introduces the THUD Robotic Dataset, which stands out from prior benchmarks due to its explicit focus on dynamic complexity:
- Scale: 90,175 annotated frames (84,984 synthetic, 5,191 real-world).
- Scenarios: 13 large-scale indoor environments (8 real-world campus scenes like canteens, laboratories, and cafes; 5 synthetic scenes like supermarkets and offices) with an average area exceeding 300 $m^2$.
- Annotations: Over 20 million total labels, including 1.2 million labels specifically tracking dynamic objects (pedestrians, robots, shopping carts) across 91 object categories.
Key Insights and Ideas
- The Dynamic Performance Gap: Standard 3D object detection algorithms show a massive performance drop when switching from static to dynamic objects. For example, the DeMF algorithm drops from 45.43 mAP on static objects down to 28.56 mAP on dynamic objects in the canteen scene.
- Density Compounds the Challenge: The authors introduced a "dynamic complexity" metric (average number of dynamic pedestrians per frame). As this complexity increases, the performance gap between static and dynamic object detection widens significantly.
- Relocalization Vulnerability: Local-feature-based relocalization methods (like FeatLoc) are incredibly sensitive to dynamic noise compared to global methods (like NetVLAD). Moving people introduce visual disturbances that corrupt the robot's trajectory tracking, causing median translation and rotation errors to spike.
- Segmentation Resilience: Interestingly, semantic segmentation algorithms (like ESANet) proved relatively resilient to dynamic objects, showing only minor drops in Mean Intersection over Union (MIoU). This indicates that pixel-wise classification is less disrupted by motion than geometric localization or bounding box estimation.
Key Results
The authors benchmarked mainstream algorithms across three core tasks on the THUD dataset:
3D Object Detection
Evaluated on F-PointNet, ImVoteNet, and DeMF. In the real-world canteen scene, DeMF achieved the highest performance but still demonstrated a stark contrast: 45.43 mAP for static objects versus 28.56 mAP for dynamic objects.
Semantic Segmentation
Tested using ACNet, RedNet, ESANet, and SA-Gate. While SA-Gate performed exceptionally well on the synthetic supermarket scene (83.19% MIoU), its performance dropped sharply on the real-world canteen scene (58.34% MIoU), highlighting a significant reality gap.
Robot Relocalization
Tested using NetVLAD and FeatLoc. In crowded real-world scenes, tracking errors spiked dramatically. FeatLoc, for instance, suffered a median translation error of 2.41 meters and a rotation error of 32.4° in the canteen environment due to pedestrian interference.
Limitations and Future Work
While THUD represents a major step forward, the authors note a few limitations:
- Real-World Volume: The real-world portion of the dataset (5,191 frames) is still relatively small compared to the synthetic data (nearly 85K frames).
- Hardware Instability: Due to the inherent limitations of the Kinect V2 sensor, the real-world frame rate fluctuated between 15–30 fps, which can introduce temporal jitter.
Future Work: The team plans to continuously expand the dataset to include more crowded real-world scenarios. They also aim to extend the benchmark to support downstream dynamic tasks such as multi-object tracking, trajectory prediction, and online robotic navigation.
Summary updated