CLIPSwarm: Converting text into formations of robots

Pablo Pueyo, Eduardo Montijano, Ana C. Murillo, Mac SchwagerarXiv 2023 · DOI · arXiv

Abstract

CLIPSwarm is a novel framework that bridges the gap between high-level human intent and low-level robot control by generating swarm formations from natural language descriptions. By integrating the CLIP foundation model with a Monte Carlo particle filter, the system iteratively optimizes robot positions to match textual prompts. This approach enables zero-shot artistic expression in robotics, automating the design process for applications like drone light shows.

Introduction

Creating visually appealing robot swarm formations, such as those seen in high-tech drone light shows, is currently a labor-intensive process. Human designers must manually specify coordinates or use specialized CAD tools to define patterns. CLIPSwarm addresses this by allowing users to describe a shape in natural language (e.g., "the contour of a drop") and automatically translating that intent into spatial coordinates for a robot swarm.

The CLIPSwarm Framework

The core of CLIPSwarm is an iterative optimization pipeline that leverages the visual-linguistic understanding of the CLIP (Contrastive Language-Image Pre-training) model.

The CLIPSwarm architecture uses an iterative loop involving CLIP similarity to refine robot formations.

The process follows five key steps:

  1. Initialization: The system begins with random robot formations.
  2. Visual Representation: To provide visual context, the algorithm calculates a Convex-Hull around the robots, transforming a set of discrete points into a recognizable silhouette.
  3. CLIP Metric: The CLIP model evaluates the similarity between the user's text prompt and the generated silhouette.
  4. Iterative Refinement: Using a Monte Carlo particle filter, the system selects the best-performing formations and introduces slight perturbations to robot positions to improve the similarity score.
  5. Simulation: Optimized coordinates are verified in AirSim, a photorealistic drone simulator integrated with ROS, ensuring the formations are achievable by physical hardware.

Results and Performance

The authors evaluated CLIPSwarm across several test cases, including "a shape of a diamond," "a circle outline," and "the contour of a drop." In all instances, the CLIP similarity score increased significantly over 50 iterations, indicating successful convergence toward the target shape.

Comparison between the optimized point-cloud contour and the resulting drone formation for "The contour of a drop."

For the "drop" test case, the similarity score improved from 0.271 to 0.312. Qualitatively, the system successfully generated recognizable shapes that were then mapped to 70 drones in a simulated environment, proving the pipeline's end-to-end viability for real-world swarm robotics.

Discussion and Limitations

While CLIPSwarm demonstrates a powerful zero-shot capability for artistic robotics, it faces certain constraints. The current reliance on Convex-Hulls prevents the representation of concave shapes (like stars) or internal details. Furthermore, CLIP's "opinion" can sometimes favor simpler geometric shapes over detailed ones; for example, preferring a pentagon over a detailed house icon. Future work aims to expand the system to 3D volumetric formations and explore more efficient optimization methods to increase speed.

Summary updated

BibTeX
@misc{pueyo2023clipswarm,
  author = {Pablo Pueyo and Eduardo Montijano and Ana C. Murillo and Mac Schwager},
  title = {{CLIPSwarm: Converting text into formations of robots}},
  year = {2023},
  eprint = {2311.11047},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11047},
}
Figure