DREAM: Decentralized Reinforcement Learning for Exploration and Efficient Energy Management in Multi-Robot Systems

Dipam Patel, Phu Pham, Kshitij Tiwari, Aniket BeraarXiv 2023 · DOI · arXiv

Abstract

The DREAM framework (Decentralized Reinforcement Learning for Exploration and Efficient Energy Management) addresses the critical conflict between exploration efficiency and battery constraints in multi-robot systems (MRS). By integrating a refined navigation model (RTD3) with a Graph Neural Network (GNN) for energy-aware goal allocation, the system achieves a ~25% reduction in energy consumption and a 75% reduction in model parameters compared to existing baselines. This paper details the architecture, training methodology, and performance of DREAM in simulated resource-constrained environments.

Introduction

In multi-robot systems, two primary objectives often conflict: maximizing exploration efficiency and adhering to strict energy constraints. Conventional systems frequently rely on heuristic-based task planning or random goal allocation. These methods often lead to "stranded" robots—agents that deplete their power reserves before completing a mission or returning to a charging station.

Furthermore, many deep reinforcement learning (DRL) models for navigation are computationally intensive, making them impractical for the low-power hardware typically found on small-scale mobile robots. The DREAM framework bridges this gap by providing a decentralized navigation strategy that allows robots to make independent decisions while coordinating goal assignments through a centralized logic to minimize total team energy expenditure.

Methodology

The DREAM framework utilizes a dual-layered approach to manage navigation and task allocation.

1. Refined TD3 (RTD3) for Navigation

The navigation layer is built upon a modified Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, optimized for obstacle avoidance and goal-seeking. A key innovation in this layer is the Reward Categorized Replay Buffer (RCRB). Unlike standard buffers that sample experiences uniformly, the RCRB sorts experiences into Positive, Neutral, and Negative buffers. This ensures the model learns equally from rare successes and common failures, significantly accelerating convergence.

The Reward Categorized Replay Buffer (RCRB) architecture used to balance training experiences.

Additionally, the agents undergo Curriculum Learning, starting in simple, obstacle-free environments and progressing to complex scenarios with randomized start/end points and dense obstacles.

2. GNN for Energy Management

A Graph Neural Network (GNN) is employed to solve the "assignment problem." The GNN evaluates the global state of the MRS—including robot positions, battery levels, and goal locations—to assign targets that minimize cumulative energy use. Realistic battery discharge is simulated using the Doyle-Fuller-Newman (DFN) model via the PyBaMM library.

Key Results and Discussion

The DREAM framework was evaluated in five distinct simulated environments using ROS2 and Gazebo.

Energy Efficiency and Mission Success

The GNN-based goal allocation achieved an average energy reduction of 21% to 25% compared to random allocation. By penalizing assignments that would likely result in stranded robots, the network favored solutions where every agent maintained sufficient battery to reach its goal and return home.

Comparison between Random Goal Allocation and the proposed GNN-based approach, highlighting improved energy distribution and higher remaining battery levels.

Model Optimization

By streamlining the Actor network architecture, the authors reduced the number of model parameters by 75% compared to previous state-of-the-art models. This reduction is vital for deploying AI on resource-constrained hardware without sacrificing navigation performance. The model also demonstrated resilience in dynamic environments, successfully treating other moving robots as dynamic obstacles.

Conclusion and Future Work

DREAM provides a robust, energy-efficient solution for multi-robot exploration. While the current results are promising in simulation, future work will focus on physical robot experiments to validate the policy's transferability. Additionally, the authors aim to incorporate mechanisms for handling hardware failures, allowing remaining agents to dynamically redistribute tasks if a teammate becomes incapacitated.

Summary updated

Abstract

Resource-constrained robots often suffer from energy inefficiencies, underutilized computational abilities due to inadequate task allocation, and a lack of robustness in dynamic environments, all of which strongly affect their performance. This paper introduces DREAM - Decentralized Reinforcement Learning for Exploration and Efficient Energy Management in Multi-Robot Systems, a comprehensive framework that optimizes the allocation of resources for efficient exploration. It advances beyond conventional heuristic-based task planning as observed conventionally. The framework incorporates Operational Range Estimation using Reinforcement Learning to perform exploration and obstacle avoidance in unfamiliar terrains. DREAM further introduces an Energy Consumption Model for goal allocation, thereby ensuring mission completion under constrained resources using a Graph Neural Network. This approach also ensures that the entire Multi-Robot System can survive for an extended period of time for further missions compared to the conventional approach of randomly allocating goals, which compromises one or more agents. Our approach adapts to prioritizing agents in real-time, showcasing remarkable resilience against dynamic environments. This robust solution was evaluated in various simulated environments, demonstrating adaptability and applicability across diverse scenarios. We observed a substantial improvement of about 25% over the baseline method, leading the way for future research in resource-constrained robotics.
BibTeX
@misc{patel2023dream,
  author = {Dipam Patel and Phu Pham and Kshitij Tiwari and Aniket Bera},
  title = {{DREAM: Decentralized Reinforcement Learning for Exploration and Efficient Energy Management in Multi-Robot Systems}},
  year = {2023},
  eprint = {2309.17433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.17433},
}
Figure