Leveraging Human Feedback to Evolve and Discover Novel Emergent Behaviors in Robot Swarms

Connor Mattson, Daniel S. Brown

Abstract

Predicting and discovering emergent behaviors in robot swarms remains a significant challenge due to the complex relationship between local agent rules and global collective patterns. Traditionally, researchers have relied on hand-engineered features to categorize these behaviors—a process that is labor-intensive and often fails to scale with hardware complexity. This paper introduces a novel framework that automates the discovery of swarm behaviors by combining self-supervised deep learning with human-in-the-loop (HIL) refinement. By treating swarm trajectories as visual data, the system learns a latent behavior space that outperforms expert-defined metrics, ultimately discovering two previously unknown emergent patterns in multi-robot systems.

Introduction

Robot swarms are characterized by emergence: the phenomenon where simple, local interactions among individual agents result in complex, large-scale collective behaviors such as flocking, milling, or clustering. While these systems are robust and scalable, they are notoriously difficult to design and analyze. Historically, identifying the "menu" of possible behaviors for a given robot hardware configuration required domain experts to manually define mathematical features (e.g., scatter, angular momentum, or group velocity).

This manual bottleneck limits the exploration of more complex robotic platforms. To address this, we propose a pipeline that eliminates the need for expert-crafted features. Instead, our system learns to perceive and categorize "interesting" behaviors directly from raw simulation data, guided by a small amount of human intuition to align the machine's latent space with human conceptual categories.

Methodology

The proposed framework maps the vast space of possible robot controllers into a low-dimensional "latent behavior space" through a multi-stage pipeline:

1. Visual Representation and Self-Supervised Learning

To create a universal language for swarm behavior, the system collapses the final frames of swarm trajectories into 50x50 grayscale images. This transformation allows the use of computer vision techniques to analyze robotic movement. We employ a triplet loss architecture for self-supervised pretraining. The model learns to group similar behaviors by comparing "synthetic" pairs—taking a single swarm image and creating a positive version through random rotations and crops, while contrasting it against a different simulation as a negative example.

2. Human-in-the-Loop (HIL) Refinement

While self-supervised learning captures visual similarity, it may not always align with what a human researcher considers a distinct "behavior." We introduce a refinement phase where a human user provides labels for a small fraction of the data. This feedback adjusts the latent space, ensuring that the distance between points in the embedding reflects meaningful behavioral differences.

To optimize the search process, we implement heuristic filters to prune "uninteresting" controllers—specifically those resulting in static agents (Passive) or agents that ignore their neighbors (Neglectful). This reduces the search space by over 20%. Finally, we use Novelty Search to evolve new controllers that target unexplored regions of the learned latent space, followed by k-medoids clustering to organize the results into a final taxonomy.

Experimental Results

Performance Benchmarks

We evaluated the framework using a custom Python-based simulator across two robot capability models: a standard single-sensor model and a more complex two-sensor model.

  • Accuracy: The HIL-trained latent embedding achieved 88.09% accuracy in grouping similar behaviors, significantly outperforming random initialization (69%) and even expert-crafted features (84%).
  • Discovery Rate: On the single-sensor model, our automated approach discovered 3% more distinct behaviors than the previous state-of-the-art. On the complex two-sensor model, the improvement jumped to 19%.

Discovery of Novel Behaviors

The system successfully identified two entirely new emergent behaviors for computation-free swarms:

  1. Nested Cycle: A multi-layered behavior where an outer perimeter of agents effectively traps an inner ring of agents into a milling pattern.
  2. Concave Path: A wave-like cyclic pattern where agents form a non-convex, oscillating loop.

Discussion and Conclusion

Our results demonstrate that raw trajectory images are sufficient to learn a robust behavior space for robot swarms. By moving from one sensor to two, the potential behavior space grows exponentially (to approximately $3.77 \times 10^{12}$ possible controllers), enabling the emergence of non-convex and multi-layered patterns previously thought impossible for simple agents.

However, limitations remain. The visual approach may struggle with behaviors defined by internal states or non-visual signals (e.g., wireless communication). Furthermore, extremely rare behaviors like the Concave Path remain difficult for automated search algorithms to find consistently. Future work will focus on incorporating multi-modal data—such as signal strength or acoustic data—into the latent space to capture an even broader spectrum of robotic intelligence.

Summary updated

BibTeX
@inproceedings{mattson2023leveraging,
  author = {Connor Mattson and Daniel S. Brown},
  title = {{Leveraging Human Feedback to Evolve and Discover Novel Emergent Behaviors in Robot Swarms}},
  booktitle = {GECCO},
  year = {2023},
  doi = {10.1145/3583131.3590443},
}
Figure