Intrinsic-motivation multi-robot social formation navigation with coordinated exploration

Hao Fu, Wei Liu, Shuai Zhou

TL;DR

This paper introduces CEMRRL, a novel reinforcement learning algorithm for multi-robot formation navigation in human-populated environments. CEMRRL tackles the challenge of unpredictable pedestrian behavior by employing a self-learning intrinsic reward mechanism that encourages coordinated exploration, leading to more efficient and safer navigation.

What Problem Does This Paper Address?

Navigating in environments with humans, like sidewalks or public spaces, is a significant challenge for multi-robot systems. These environments are dynamic and unpredictable due to human movement. While reinforcement learning (RL) offers a promising approach for robot navigation, existing methods struggle with:

  • Unpredictable Pedestrian Dynamics: Pedestrians don't always follow predictable patterns, making it hard for robots to plan safe and efficient paths.
  • Coordinated Exploration: Robots need to explore the environment effectively to learn optimal navigation strategies, but uncoordinated exploration can lead to redundant efforts and slow learning.
  • Policy Conservatism: To ensure safety, robots might adopt overly cautious behaviors, leading to slower navigation and less efficient formation maintenance.

This paper specifically addresses the problem of social formation navigation, where multiple robots must maintain a specific formation while navigating through environments populated by humans. The core challenge lies in achieving efficient and coordinated exploration that leads to robust and adaptive navigation policies, even in the face of unpredictable human behavior.

Methods

The paper proposes the Coordinated-Exploration Multi-Robot Reinforcement Learning (CEMRRL) algorithm. Its key innovations include:

  • Self-Learning Intrinsic Reward Mechanism: This is the core of CEMRRL. It's designed to encourage robots to explore novel states and generate diverse trajectories. It combines:
  • Life-long Novelty Function ($N_d$): Rewards robots for visiting less frequently encountered joint states.
  • Episodic Exploration Bonus ($b_s$): Encourages exploration within an episode, incorporating a novelty differential function to promote coordinated exploration.
  • Joint Policy Entropy ($c_s$): Promotes diversity in the joint action distribution, preventing premature convergence to myopic behaviors.
  • Dual-Sampling Mode: Used within a Centralized Training and Decentralized Execution (CTDE) framework. This mode enhances the representation of both the navigation policy and the intrinsic reward.
  • Two-Time-Scale Update Rule: This rule decouples the updates for the intrinsic reward parameters and the actor-critic parameters. The intrinsic reward parameters are updated at a faster rate (per timestep), while actor-critic parameters are updated at a slower rate (per episode). This helps to mitigate issues arising from the actor-critic learning lagging behind the intrinsic reward learning.

Datasets

The paper does not introduce new datasets. Instead, it uses a simulated environment created in Python for evaluating the algorithm. The simulation includes:

  • A team of robots (one leader, two followers) configured for a triangle formation.
  • 5 pedestrians, randomly generated and controlled using the ORCA model.
  • Gaussian noise added to observations and actions to enhance robustness.

The experiments are conducted across three random testing scenarios with varying numbers of pedestrians (5, 7, and 9).

Key Insights and Ideas

  • Intrinsic Motivation for Coordination: The paper demonstrates that intrinsic motivation, when designed to encourage coordinated exploration, can significantly improve multi-robot navigation in complex social environments. It's not just about individual exploration, but how robots explore together.
  • Bridging the Exploration Gap: The intrinsic reward mechanism effectively bridges the gap left by sparse extrinsic rewards, which are common in social navigation tasks. This leads to more efficient learning and better policy development.
  • Decoupling for Stability: The two-time-scale update rule is crucial for stabilizing the learning process. By updating intrinsic reward parameters more frequently, the system can adapt faster to environmental changes, while the slower actor-critic updates ensure gradual policy improvement.
  • Mitigating Policy Conservatism: The proposed intrinsic reward mechanism helps robots overcome overly conservative behaviors by encouraging them to explore more diverse strategies, leading to faster and more direct navigation.

Key Results

  • Superior Performance: CEMRRL consistently outperforms baseline methods (MR-SAC, MR-Att-RL) across key metrics like success rate, collision rate, and navigation time in various pedestrian density scenarios.
  • Faster Convergence: CEMRRL achieves convergence approximately 3 times faster than MR-SAC (20,000 episodes vs. 65,000 episodes).
  • Improved Safety: The algorithm maintains a greater minimum distance to collision, indicating safer navigation.
  • Extensibility: The integration of a self-attention mechanism (CEMRRL-Att) further improves performance, demonstrating the algorithm's extensibility.
  • Robustness: CEMRRL shows resilience to unprecedented pedestrian behaviors like sudden stops or rapid direction changes, maintaining formation and successfully navigating crowds.

Limitations and Future Work

  • Parameter Coupling: The authors mention that in future work, they will focus on analyzing the convergence of their proposed algorithm to better quantify the effect of parameter coupling.
  • Computational Intensity: While not explicitly stated as a limitation, the paper mentions that direct application of evolutionary algorithms to online multi-robot navigation can be computationally intensive, implying that their RL-based approach aims to address this.
  • Simulation-to-Real Gap: As with most simulation-based research, the direct transferability of these results to real-world robots might require further investigation and adaptation.

Summary updated

Abstract

This paper investigates the application of reinforcement learning (RL) to multi-robot social formation navigation, a critical capability for enabling seamless human-robot coexistence. While RL offers a promising paradigm, the inherent unpredictability and often uncooperative dynamics of pedestrian behavior pose substantial challenges, particularly concerning the efficiency of coordinated exploration among robots. To address this, we propose a novel coordinated-exploration multi-robot RL algorithm introducing an intrinsic motivation exploration. Its core component is a self-learning intrinsic reward mechanism designed to collectively alleviate policy conservatism. Moreover, this algorithm incorporates a dual-sampling mode within the centralized training and decentralized execution framework to enhance the representation of both the navigation policy and the intrinsic reward, leveraging a two-time-scale update rule to decouple parameter updates. Empirical results on social formation navigation benchmarks demonstrate the proposed algorithm's superior performance over existing state-of-the-art methods across crucial metrics. Our code and video demos are available at: https://github.com/czxhunzi/CEMRRL.
BibTeX
@article{fu2025intrinsic,
  author = {Hao Fu and Wei Liu and Shuai Zhou},
  title = {{Intrinsic-motivation multi-robot social formation navigation with coordinated exploration}},
  journal = {Engineering Applications of AI},
  year = {2025},
}
Figure