Bio-inspired Dynamic and Decentralized Online Learning in Uninformed Heterogeneous Multi-Agent Environments

Angel SylvesterIJCAI 2024

Introduction

In critical applications such as search-and-rescue, environmental monitoring, or hazardous zone exploration, robotic swarms must often operate in "uninformed" environments. These are settings where prior maps are unavailable, and the distribution of targets or obstacles is unknown and potentially dynamic. Traditional machine learning paradigms often struggle here because they typically require a distinct, computationally intensive training phase in a simulator before deployment. Furthermore, many existing swarm robotics approaches rely on centralized control, which introduces single points of failure and limits scalability in communication-constrained environments.

To address these challenges, this research introduces ADAPT-GA (Adaptive Dynamic Agent Parameter Tuning Genetic Algorithm). ADAPT-GA is a decentralized framework that enables robots to learn and adapt while performing their tasks, effectively merging the training and execution phases into a continuous, online loop.

The ADAPT-GA Framework

The core of ADAPT-GA lies in its dual-layered approach to adaptation, combining bio-inspired evolutionary principles with statistical strategy selection.

  1. Embodied Evolution (EE): Unlike traditional evolutionary robotics, which evolves controllers offline, Embodied Evolution occurs in parallel across the physical swarm during deployment. Robots function as individual "genomes" within a larger population. When two robots come into local communication range (an "encounter"), they exchange genetic information—parameters such as movement speed, reward sensitivity, and exploration penalties. This peer-to-peer exchange allows successful traits to propagate through the swarm without any centralized oversight.
  2. Bayesian Strategy Selection: To navigate effectively, robots choose from a set of predefined movement strategies, including Forward Persistence, Circular Persistence, and Correlated Random Walks. The selection is governed by a Bayesian-inspired weighting system. Each robot maintains a probability distribution over these strategies, which is updated instantaneously based on performance metrics like energy consumption and task success.
The ADAPT-GA framework integrates Bayesian strategy selection with decentralized genetic algorithm parameter updates.

As illustrated in Figure 1, the strategy weights ($w$) are adjusted using a simple yet effective formula: $w(s \in S) = 1 + c \cdot (n_f + n_s)$, where $n_f$ and $n_s$ represent the number of failures and successes, respectively. This mechanism allows robots to develop an "intuition" for which movement patterns are most effective in the current environment, favoring successful behaviors while remaining flexible enough to resample strategies when conditions change.

Experimental Evaluation

The ADAPT-GA framework was evaluated using high-fidelity simulations across three distinct environmental configurations designed to mimic real-world complexity:

  • Randomly Dispersed Objects: Assessing basic search efficiency and coverage.
  • Object Clusters: Testing the swarm's ability to identify and exploit "hotspots" of high target density.
  • Obstacle-Laden Urban Settings: Evaluating navigation and adaptability in constrained, complex geometries.

These benchmarks provide a rigorous testing ground for decentralized learning, moving beyond simple obstacle avoidance to complex foraging tasks.

Key Results and Insights

The study yielded several critical insights into the performance of decentralized online learning:

  • Superior Efficiency: ADAPT-GA consistently outperformed other state-of-the-art bio-inspired methods in item collection efficiency across all tested environments.
  • The Necessity of Communication: Ablation studies demonstrated that the exchange of genetic information during encounters is the primary driver of adaptation. Without this local communication, the swarm's ability to optimize its collective behavior was significantly diminished.
  • Scalability and Resilience: Notably, the system showed high resilience to congestion. Increasing robot density did not destabilize the learning process; instead, it provided more opportunities for parameter exchange, potentially accelerating the evolutionary process.
  • Lightweight Adaptation: By using Bayesian updates instead of extensive long-term memory, the system remains computationally lightweight, making it suitable for deployment on resource-constrained robotic hardware.

Conclusion and Future Directions

ADAPT-GA represents a significant step toward truly autonomous, self-optimizing robotic swarms. By eliminating the divide between training and execution, the framework ensures that robots remain responsive to the immediate realities of their environment.

Future work will focus on enhancing the robustness of the Bayesian selection mechanism by incorporating "drift" or a forgetting factor (gamma decay). This would prevent robots from becoming overly committed to strategies that were successful in the past but are no longer optimal. Additionally, the integration of more sophisticated path-planning techniques and the exploration of "altruistic" behaviors through Multi-Agent Reinforcement Learning (MARL) offer promising avenues for further increasing the collective intelligence of robotic swarms.

Summary updated

BibTeX
@inproceedings{sylvester2024bio,
  author = {Angel Sylvester},
  title = {{Bio-inspired Dynamic and Decentralized Online Learning in Uninformed Heterogeneous Multi-Agent Environments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure