Abstract
Simulation engines are widely adopted in robotics. However, they lack either full simulation control, ROS integration, realistic physics, or photorealism. Recently, synthetic data generation and realistic rendering has advanced tasks like target tracking and human pose estimation. However, when focusing on vision applications, there is usually a lack of information like sensor measurements or time continuity. On the other hand, simulations for most robotics tasks are performed in (semi)static environments, with specific sensors and low visual fidelity. To solve this, we introduced in our previous work a fully customizable framework for generating realistic animated dynamic environments (GRADE). We use GRADE to generate an indoor dynamic environment dataset and then compare multiple SLAM algorithms on different sequences. By doing that, we show how current research over-relies on known benchmarks, failing to generalize. Our tests with refined YOLO and Mask R-CNN models provide further evidence that additional research in dynamic SLAM is necessary.
Introduction
Simultaneous Localization and Mapping (SLAM) is a cornerstone of autonomous robotics. While significant progress has been made in static environments, real-world scenarios are inherently dynamic, populated by people, pets, and moving objects. Developing robust SLAM algorithms for these conditions is challenging due to the risks of real-world testing and the limitations of existing simulators. Many current "dynamic SLAM" methods rely on aging benchmarks like the TUM RGB-D dataset, leading to a potential generalization gap where algorithms overfit to specific dataset characteristics and fail to perform in diverse real-world conditions.
The GRADE Framework
To address these limitations, we utilize GRADE (Generating Realistic Animated Dynamic Environments). GRADE is a photorealistic, fully controllable, and ROS-integrated framework designed to bridge the gap between simulation and reality. It allows for the creation of complex, non-rigid animations and precise sensor configurations.
Using GRADE, we generated a comprehensive indoor dataset featuring:
- Static (S) and Dynamic (D) sequences: Ranging from simple static rooms to environments with multiple moving agents.
- Challenging Scenarios: Including flying objects (F) and significant camera occlusions (WO).
- Realistic Sensor Models: Incorporating simulated noise such as rolling shutter effects, depth noise, and IMU drift.
Beyond environment generation, we explored retraining perception networks (YOLOv5 and Mask R-CNN) using synthetic data from GRADE. We compared these against models trained on standard datasets like COCO, using the TUM RGB-D (fr3/walking) dataset as a real-world baseline to evaluate sim-to-real transfer.
Benchmarking and Metrics
We evaluated several prominent SLAM frameworks against the GRADE dataset, including static methods (RTABMap, ORB-SLAMv2) and dynamic-aware or learning-based methods (DynaSLAM, DynamicVINS, StaticFusion, TartanVO).
A critical contribution of our evaluation is the emphasis on the "Missing Time" metric. While Absolute Trajectory Error (ATE) is the standard benchmark, it can be misleading if an algorithm fails to track the camera for a significant portion of the sequence. A low ATE on only 10% of a trajectory does not indicate a successful SLAM system; robust performance requires high tracking duration.
Key Results and Discussion
Our benchmarks reveal a significant performance degradation when state-of-the-art algorithms are moved from standard benchmarks to the GRADE environment.
Key findings include:
- Tracking Failures: Many advanced dynamic SLAM methods struggled with continuity. For instance, DynaSLAM lost tracking for more than half of the duration in several 60-second sequences (e.g., 57.07s in the DH sequence).
- Robustness of DynamicVINS: DynamicVINS emerged as one of the more resilient methods, maintaining tracking more consistently than others, although its ATE remained high in complex dynamic scenarios like "Dynamic Horizontal" (DH).
- The Sim-to-Real Advantage: Pre-training detection networks on synthetic GRADE data yielded performance comparable to or better than models trained on real-world datasets like COCO. This demonstrates the value of high-fidelity synthetic data for perception tasks within SLAM pipelines.
Conclusion
The results obtained using the GRADE framework highlight that dynamic SLAM remains a notoriously difficult and unsolved problem. Current research's over-reliance on a limited set of benchmarks has masked a significant lack of generalization. By providing a fully customizable and photorealistic simulation tool, we hope to encourage the development of more robust algorithms that can handle the "messy reality" of dynamic environments. Future work will focus on parameter optimization, active SLAM strategies, and further narrowing the sim-to-real gap.
The code, results, and generated data are available at: https://eliabntt.github.io/grade-rr
Summary updated