Abstract
CLIPSwarm is a novel framework that bridges the gap between high-level human intent and low-level robot control by generating swarm formations from natural language descriptions. By integrating the CLIP foundation model with a Monte Carlo particle filter, the system iteratively optimizes robot positions to match textual prompts. This approach enables zero-shot artistic expression in robotics, automating the design process for applications like drone light shows.
Introduction
Creating visually appealing robot swarm formations, such as those seen in high-tech drone light shows, is currently a labor-intensive process. Human designers must manually specify coordinates or use specialized CAD tools to define patterns. CLIPSwarm addresses this by allowing users to describe a shape in natural language (e.g., "the contour of a drop") and automatically translating that intent into spatial coordinates for a robot swarm.
The CLIPSwarm Framework
The core of CLIPSwarm is an iterative optimization pipeline that leverages the visual-linguistic understanding of the CLIP (Contrastive Language-Image Pre-training) model.
The process follows five key steps:
- Initialization: The system begins with random robot formations.
- Visual Representation: To provide visual context, the algorithm calculates a Convex-Hull around the robots, transforming a set of discrete points into a recognizable silhouette.
- CLIP Metric: The CLIP model evaluates the similarity between the user's text prompt and the generated silhouette.
- Iterative Refinement: Using a Monte Carlo particle filter, the system selects the best-performing formations and introduces slight perturbations to robot positions to improve the similarity score.
- Simulation: Optimized coordinates are verified in AirSim, a photorealistic drone simulator integrated with ROS, ensuring the formations are achievable by physical hardware.
Results and Performance
The authors evaluated CLIPSwarm across several test cases, including "a shape of a diamond," "a circle outline," and "the contour of a drop." In all instances, the CLIP similarity score increased significantly over 50 iterations, indicating successful convergence toward the target shape.
For the "drop" test case, the similarity score improved from 0.271 to 0.312. Qualitatively, the system successfully generated recognizable shapes that were then mapped to 70 drones in a simulated environment, proving the pipeline's end-to-end viability for real-world swarm robotics.
Discussion and Limitations
While CLIPSwarm demonstrates a powerful zero-shot capability for artistic robotics, it faces certain constraints. The current reliance on Convex-Hulls prevents the representation of concave shapes (like stars) or internal details. Furthermore, CLIP's "opinion" can sometimes favor simpler geometric shapes over detailed ones; for example, preferring a pentagon over a detailed house icon. Future work aims to expand the system to 3D volumetric formations and explore more efficient optimization methods to increase speed.
Summary updated