Arena-Rosnav 2.0: A Development and Benchmarking Platform for Robot Navigation in Highly Dynamic Environments

Linh Kästner, Reyk Carstens, Huajian Zeng, Jacek Kmiecik, Tuan Anh Le, Teham Bhuiyan, Boris Meinardus, Jens LambrechtIROS 2023 · DOI · OpenReview

TL;DR

This paper introduces Arena-Rosnav 2.0, a significantly improved and modular platform for developing and benchmarking robot navigation algorithms, especially in complex, dynamic environments. It addresses the challenges of difficult setup, limited extensibility, and poor comparability in existing tools by offering a unified API, a simplified installation, and a broader range of integrated simulators, planners, and robot models.

What Problem Does This Paper Address?

Navigating in dynamic environments, where unexpected obstacles and moving agents are common, is crucial for robots in applications like logistics and human-robot collaboration. While Deep Reinforcement Learning (DRL) has shown promise, developing and comparing these approaches is hindered by several factors:

  • High Barrier to Entry: Setting up and configuring existing platforms can be complex and time-consuming, requiring specialized knowledge and hardware.
  • Limited Comparability: Benchmarks often focus on specific paradigms (e.g., only DRL or only classical planners) or lack the flexibility to compare across them.
  • Poor Extensibility: Adding new simulators, planners, or functionalities to existing platforms is often difficult, limiting research innovation.
  • Lack of Realistic Simulation: Many platforms use simplified simulations that don't accurately reflect the complexities of real-world dynamic environments, especially concerning pedestrian behavior.

These issues make it challenging for researchers to develop robust navigation systems and to reliably compare the performance of different approaches.

Methods

Arena-Rosnav 2.0 is a fundamental redesign of its predecessor, Arena-Bench. The core innovation is its modular architecture, where each component (simulators, planners, tools, etc.) is an independent GitHub repository. This allows for:

  • Simplified Installation: Users can install only the modules they need, drastically reducing setup time and repository size. Installation is now a single command.
  • Enhanced Extensibility: A unified API is provided for integrating new simulators, planners, and functionalities. This makes it much easier for researchers to contribute their own modules.
  • Unified Task Management: A central task generator handles the instantiation of maps, scenarios, and robots across different simulators, ensuring cross-compatibility.
  • Improved Simulation: The platform now includes a photorealistic Unity simulator and a resource-efficient 2D simulator. Pedestrian behavior has been made more realistic through the integration of social navigation modules.
  • Broader Planner Integration: The platform integrates a wide array of state-of-the-art navigation planners, including those from the BARN challenge and other learning-based approaches.
  • Enhanced Evaluation: A reworked evaluation class offers more customization options for plotting and metrics, including safety, efficiency, robustness, and smoothness. New task modes are also introduced, such as scenarios where the robot's exact position or the map is unknown.
  • Multi-Agent Support: The platform now supports the deployment of multiple robots and independent navigation stacks, enabling Multi-Agent Reinforcement Learning (MARL) research.

Datasets

The paper does not explicitly mention specific datasets for training DRL agents. However, it describes the use of the AWS warehouse world for qualitative evaluations. For quantitative evaluations, planners were tested on a map with varying numbers of randomly generated obstacles (5 and 10) and pedestrians. The platform itself can generate an infinite number of train and test environments through its task generator.

Key Insights and Ideas

  • Modularity is Key: The shift to a fully modular, repository-per-component architecture is the most significant advancement, directly addressing issues of installation complexity and extensibility.
  • Unified API Lowers Barrier: Providing a clear, well-documented API for integrating new components is crucial for fostering community contributions and expanding the platform's capabilities.
  • Realistic Pedestrian Behavior Matters: The integration of more sophisticated pedestrian models is vital for developing robots that can safely and effectively navigate in human-centric environments.
  • Benchmarking Across Paradigms: The platform's ability to compare classical, hybrid, and learning-based planners side-by-side is essential for understanding the trade-offs and identifying the best approaches for different scenarios.
  • User Feedback Drives Improvement: The iterative development process, incorporating feedback from user studies and GitHub issues, has led to a more robust and user-friendly system.

Key Results

  • Installation Time Reduction: Installation time for the platform has been reduced from approximately 21.5 minutes to under 2 minutes.
  • Repository Size Compression: The core repository size has been compressed from over 5.7 GB to less than 0.6 GB.
  • User Study Success: A user study with 8 researchers indicated positive feedback, with users highlighting the easy installation, good documentation, and the improved code base. Several researchers are already using the platform for their work.
  • Planner Performance Insights: Quantitative evaluations showed that hybrid planners generally performed well in terms of smoothness, while efficiency varied. Qualitative evaluations revealed that some planners struggled with high obstacle densities and exhibited jerky movements or roundabout paths.
  • New Features: Integration of two new simulators (Unity, Arena2D), several new planners (including BARN challenge participants), more robot models, and new task modes (unknown position/map).

Limitations and Future Work

  • Hardware Requirements for 3D Training: While 3D simulation (Unity) is integrated, profound training in these environments was limited due to hardware constraints. Future work will focus on leveraging more powerful hardware for this.
  • Conflicting Package Versions: Some users encountered issues with conflicting package versions for newly integrated hybrid planners. The authors have addressed this by clarifying requirements in their documentation.
  • Qualitative Plotting Complexity: Visualizing too many planners on a single qualitative plot can become complex. The authors recommend plotting no more than four planners at once.

Future work includes:

  • Incorporating more simulators and benchmarking them.
  • Comparing different sensor modalities.
  • Developing a real robot automation benchmarking pipeline.
  • Providing a web-based user interface for easier access.
  • Serving as a platform for conducting future challenges and benchmarks, with ongoing discussions with the BARN organizers.

Summary updated

Abstract

Following up on our previous works, in this paper, we present Arena-Rosnav 2.0 an extension to our previous works Arena-Bench and Arena-Rosnav, which adds a variety of additional modules for developing and benchmarking robotic navigation approaches. The platform is fundamentally restructured and provides unified APIs to add additional functionalities such as planning algorithms, simulators, or evaluation functionalities. We have included more realistic simulation and pedestrian behavior and provide a profound documentation to lower the entry barrier. We evaluated our system by first, conducting a user study in which we asked experienced researchers as well as new practitioners and students to test our system. The feedback was mostly positive and a high number of participants are utilizing our system for other research endeavors. Finally, we demonstrate the feasibility of our system by integrating two new simulators and a variety of state of the art navigation approaches and benchmark them against one another. The platform is openly available at https://github.com/Arena-Rosnav.
BibTeX
@inproceedings{kastner2023arena,
  author = {Linh Kästner and Reyk Carstens and Huajian Zeng and Jacek Kmiecik and Tuan Anh Le and Teham Bhuiyan and Boris Meinardus and Jens Lambrecht},
  title = {{Arena-Rosnav 2.0: A Development and Benchmarking Platform for Robot Navigation in Highly Dynamic Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342152},
}
Figure