Abstract
Large Language Models (LLMs) have demonstrated significant potential in autonomous robot task planning. However, existing research primarily focuses on single-robot or homogeneous multi-robot systems. Real-world scenarios often require the collaboration of heterogeneous robots with diverse physical capabilities and observation spaces. We present COHERENT, a hierarchical framework that coordinates heterogeneous teams—including drones, quadrupeds, and robotic arms—to complete complex, long-horizon tasks. COHERENT employs a centralized "Proposal-Execution-Feedback-Adjustment" (PEFA) cycle, enabling a global Task Assigner to delegate subtasks while individual Robot Executors provide feedback. Our results show that COHERENT significantly outperforms decentralized methods, achieving a 97.5% success rate in complex simulated environments and demonstrating robust performance in real-world kitchen scenarios.
Introduction
The primary challenge in multi-robot systems is managing heterogeneity. Different robots possess distinct action spaces and sensory inputs. For instance, a robotic dog can navigate floors but cannot reach high shelves, whereas a robotic arm can manipulate objects but lacks mobility. Previous decentralized approaches, where robots negotiate plans through peer-to-peer dialogue, often suffer from "hallucinations" or repetitive loops due to the lack of a unified global state.
COHERENT addresses these issues by introducing a centralized hierarchical structure that maintains a global view of the mission while leveraging the specialized capabilities of each agent.
Methodology
The core of COHERENT is the PEFA (Proposal-Execution-Feedback-Adjustment) cycle, which facilitates structured communication between a centralized "brain" and individual robot "limbs."
- Proposal: A centralized Task Assigner LLM receives high-level human instructions and environmental data. It decomposes the task into subtasks and assigns them to specific robots based on their known capabilities (e.g., "Robotic Dog: Take the apple to the quadrotor's basket").
- Execution: Each robot is equipped with a Robot Executor LLM. It receives its assigned subtask, observes its local environment, and selects specific commands from a predefined action list (e.g.,
[grab] <apple>). - Feedback: Upon attempting an action, the robot performs self-reflection. It reports success or failure back to the Assigner. Crucially, it provides semantic reasons for failure (e.g., "The table is too high"), allowing for intelligent replanning.
- Adjustment: The Task Assigner updates the global plan based on the feedback and a persistent Dialogue History. This history ensures the system maintains context over long-horizon tasks, preventing the "forgetting" common in simpler LLM planners.
Experimental Setup
To evaluate COHERENT, we developed a benchmark based on the BEHAVIOR-1K simulation environment. It consists of 100 complex tasks across five realistic scenes (apartments, grocery stores, restaurants). Tasks are categorized by the number of robot types required: Mono-type, Dual-type, and Trio-type (the most complex, requiring drones, quadrupeds, and arms).
Results and Discussion
COHERENT demonstrates superior performance across all metrics compared to decentralized dialogue models (DMRS) and Monte Carlo Tree Search (MCTS) baselines.
- Success Rate (SR): COHERENT achieved an overall success rate of 97.5%, significantly higher than the best baseline (CMRS at 62.5%). Notably, it maintained a 100% success rate on the most complex Trio-type tasks, where other planners often failed completely.
- Efficiency: The system is highly efficient, requiring an average of 12.4 steps per task, closely approaching the human-designed ground truth of 10.3 steps.
- Centralization Advantage: The results confirm that a centralized Task Assigner is more effective for heterogeneous teams than decentralized dialogue, as it prevents conflicting assumptions and maintains a coherent global state.
- Real-World Validation: The framework was successfully validated in a physical kitchen environment. A robotic dog opened a door for a swarm of quadrotors, which then collaborated with a robotic arm to retrieve items.
Conclusion
While COHERENT is highly effective, it faces challenges regarding token efficiency as dialogue history grows. Additionally, reliance on cloud-based LLMs introduces latency that may impact performance in highly dynamic environments. Future work will explore dynamic action discovery and optimizing communication overhead for even longer-horizon missions.
Summary updated