Abstract
The DREAM framework (Decentralized Reinforcement Learning for Exploration and Efficient Energy Management) addresses the critical conflict between exploration efficiency and battery constraints in multi-robot systems (MRS). By integrating a refined navigation model (RTD3) with a Graph Neural Network (GNN) for energy-aware goal allocation, the system achieves a ~25% reduction in energy consumption and a 75% reduction in model parameters compared to existing baselines. This paper details the architecture, training methodology, and performance of DREAM in simulated resource-constrained environments.
Introduction
In multi-robot systems, two primary objectives often conflict: maximizing exploration efficiency and adhering to strict energy constraints. Conventional systems frequently rely on heuristic-based task planning or random goal allocation. These methods often lead to "stranded" robots—agents that deplete their power reserves before completing a mission or returning to a charging station.
Furthermore, many deep reinforcement learning (DRL) models for navigation are computationally intensive, making them impractical for the low-power hardware typically found on small-scale mobile robots. The DREAM framework bridges this gap by providing a decentralized navigation strategy that allows robots to make independent decisions while coordinating goal assignments through a centralized logic to minimize total team energy expenditure.
Methodology
The DREAM framework utilizes a dual-layered approach to manage navigation and task allocation.
1. Refined TD3 (RTD3) for Navigation
The navigation layer is built upon a modified Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, optimized for obstacle avoidance and goal-seeking. A key innovation in this layer is the Reward Categorized Replay Buffer (RCRB). Unlike standard buffers that sample experiences uniformly, the RCRB sorts experiences into Positive, Neutral, and Negative buffers. This ensures the model learns equally from rare successes and common failures, significantly accelerating convergence.
Additionally, the agents undergo Curriculum Learning, starting in simple, obstacle-free environments and progressing to complex scenarios with randomized start/end points and dense obstacles.
2. GNN for Energy Management
A Graph Neural Network (GNN) is employed to solve the "assignment problem." The GNN evaluates the global state of the MRS—including robot positions, battery levels, and goal locations—to assign targets that minimize cumulative energy use. Realistic battery discharge is simulated using the Doyle-Fuller-Newman (DFN) model via the PyBaMM library.
Key Results and Discussion
The DREAM framework was evaluated in five distinct simulated environments using ROS2 and Gazebo.
Energy Efficiency and Mission Success
The GNN-based goal allocation achieved an average energy reduction of 21% to 25% compared to random allocation. By penalizing assignments that would likely result in stranded robots, the network favored solutions where every agent maintained sufficient battery to reach its goal and return home.
Model Optimization
By streamlining the Actor network architecture, the authors reduced the number of model parameters by 75% compared to previous state-of-the-art models. This reduction is vital for deploying AI on resource-constrained hardware without sacrificing navigation performance. The model also demonstrated resilience in dynamic environments, successfully treating other moving robots as dynamic obstacles.
Conclusion and Future Work
DREAM provides a robust, energy-efficient solution for multi-robot exploration. While the current results are promising in simulation, future work will focus on physical robot experiments to validate the policy's transferability. Additionally, the authors aim to incorporate mechanisms for handling hardware failures, allowing remaining agents to dynamically redistribute tasks if a teammate becomes incapacitated.
Summary updated