Abstract: Adaptive collaboration is critical to a team of autonomous robots to perform complicated navigation tasks in large-scale unknown environments. An effective collaboration strategy should be determined and adapted according to each robot's skills and current status to successfully achieve the shared goal. We present SayCoNav, a new approach that leverages large language models (LLMs) for automatically generating this collaboration strategy among a team of robots. Building on the collaboration strategy, each robot uses the LLM to generate its plans and actions in a decentralized way. By sharing information to each other during navigation, each robot also continuously updates its step-by-step plans accordingly. We evaluate SayCoNav on Multi-Object Navigation (MultiON) tasks, that require the team of the robots to utilize their complementary strengths to efficiently search multiple different objects in unknown environments. By validating SayCoNav with varied team compositions and conditions against baseline methods, our experimental results show that SayCoNav can improve search efficiency by at most 44.28% through effective collaboration among heterogeneous robots. It can also dynamically adapt to the changing conditions during task execution.
1. Introduction
Effective collaboration in multi-robot systems is a long-standing challenge, typically requiring centralized coordination or rigid, hand-coded rules. These approaches often fail in dynamic, unknown environments or when dealing with heterogeneous teams where robots possess diverse skills. We introduce SayCoNav, a decentralized framework that leverages the semantic reasoning capabilities of Large Language Models (LLMs) to generate and adapt collaboration strategies in real-time.
2. The SayCoNav Framework
SayCoNav employs a hierarchical decentralized architecture. Each robot operates with three distinct planning levels:
- Top-Level Global Planner: This LLM-based component generates a high-level collaboration strategy by considering the "resumes" (skills and constraints) of all team members. It can be re-triggered to adapt to changes in robot status, such as hardware degradation or battery levels.
- Middle-Level Local Planner: Also LLM-based, this planner uses the global strategy and a text-based "Scene Graph" of the robot's surroundings to formulate specific, step-by-step exploration plans (e.g., "Go to the kitchen and check the fridge").
- Bottom-Level Action Planner: This module translates high-level plans into primitive motor commands (e.g.,
move_forward,open_object) using standard pathfinding algorithms like A*.
Communication is decentralized; robots synchronize by sharing status updates and scene graph information. This allows the LLM to act as a distributed manager that understands semantic context—for instance, knowing a tomato is likely in a kitchen and might be inside a closed container like a fridge.
3. Experimental Evaluation
We evaluated SayCoNav using the Multi-Object Navigation (MultiON) task within the ProcTHOR simulator. The tasks involved finding multiple target objects in unknown house layouts with teams of robots possessing specialized skills: Fast Movement (F), Manipulation (M) for opening containers, and High-Resolution Cameras (H) for detecting small items like pens or credit cards.
4. Results and Discussion
Experimental results demonstrate that SayCoNav significantly improves search efficiency compared to non-collaborative baselines. In heterogeneous teams of three, SayCoNav reduced average episode time by 44.28% compared to the SayNav baseline.
Key insights from the study include:
- Adaptive Re-planning: The system successfully re-tasks agents when physical conditions change. For example, if a fast robot's speed is throttled mid-mission, the LLM recognizes this and shifts scouting responsibilities, improving success rates from 88% to 92% in degraded scenarios.
- Semantic Efficiency: By leveraging LLM common-sense knowledge, the system avoids exhaustive searches, directing specialized robots to high-probability areas based on the target object's nature.
- Scalability and Robustness: The decentralized nature ensures the system remains robust across different team sizes and compositions, maintaining success rates between 90% and 96%.
5. Conclusion
SayCoNav highlights the potential of LLMs to solve complex coordination problems in robotics. By providing a semantic layer for reasoning and communication, it enables heterogeneous teams to collaborate adaptively in unknown environments. Future work will focus on addressing LLM hallucinations and deploying the system on physical hardware to test its resilience against real-world sensor noise and communication delays.
Summary updated