MALMM: Multi-Agent Large Language Models for Zero-Shot Robotic Manipulation

Harsh Singh, Rocktim Jyoti Das, Mingfei Han, Preslav Nakov, Ivan Laptev

TL;DR

This paper introduces MALMM, a novel multi-agent Large Language Model (LLM) framework for zero-shot robotic manipulation. By distributing planning across specialized agents (Planner, Coder, Supervisor) and incorporating real-time environmental feedback, MALMM significantly outperforms single-agent LLM approaches in complex, long-horizon tasks and demonstrates robust failure recovery.

What Problem Does This Paper Address?

While LLMs have shown impressive capabilities in planning and reasoning, their application to robotic manipulation, especially for complex, long-horizon tasks, faces significant hurdles. Existing LLM-based methods often struggle with:

  • Hallucinations: LLMs can generate incorrect plans or disregard geometric constraints, leading to task failures.
  • Failure Recovery: They lack robust mechanisms to detect and adapt to intermediate failures during execution.
  • Generalization: Many methods rely on pre-trained skill policies or in-context learning, limiting their ability to generalize to unseen tasks.
  • Long-Horizon Tasks: The complexity of extended tasks can overwhelm single LLM agents, leading to a loss of context and errors.

This paper aims to address these limitations by proposing a more robust and adaptable LLM framework for robotic manipulation that can handle complex, unseen tasks without prior task-specific training.

Methods

The core innovation is the Multi-Agent Large Language Model for Manipulation (MALMM) framework. It comprises three specialized LLM agents and a code executor:

  • Planner Agent: Responsible for breaking down the natural language task instruction into a sequence of high-level manipulation steps. It also re-evaluates the plan based on intermediate environment observations.
  • Coder Agent: Translates the Planner's steps into executable Python code for robot control.
  • Supervisor Agent: Acts as the orchestrator, managing the transitions between the Planner, Coder, and the code executor. It decides which agent to activate next based on the task, environment state, and conversation history.
  • Code Executor: Executes the generated Python code in the robotic environment.

A key aspect of MALMM is its ability to incorporate environment observations after each step. This allows the framework to detect failures (e.g., collisions, missed grasps) and trigger adaptive re-planning, significantly improving robustness. Unlike prior work, MALMM does not rely on pre-trained skill policies or in-context learning examples, enabling true zero-shot generalization.

Datasets

The evaluation was conducted using the RLBench benchmark, which provides a suite of language-conditioned robotic manipulation tasks. The authors sampled 9 RLBench tasks with 25 samples per task, featuring diverse object poses, shapes, and task horizons. They also performed experiments on five real-world tasks using a Franka robot arm.

Key Insights and Ideas

  • Specialization is Key: Distributing planning and control across specialized LLM agents (Planner, Coder) significantly improves performance and reduces hallucinations compared to a single, monolithic LLM.
  • Feedback Loop is Crucial: Incorporating intermediate environment observations after each action is vital for failure detection and adaptive re-planning, especially in long-horizon tasks.
  • Supervisor Orchestration: A dedicated supervisor agent is essential for managing the complex interactions between multiple LLM agents and ensuring a coherent workflow.
  • Zero-Shot Generalization: The framework's ability to perform unseen tasks without task-specific fine-tuning or in-context examples is a significant step towards more versatile robotic agents.
  • Failure Recovery is Achievable: MALMM demonstrates a remarkable ability to recover from dropped objects and other execution errors, a common failure point for single-agent LLM approaches.

Key Results

  • Outperforms Baselines: MALMM consistently outperformed existing zero-shot LLM-based methods (CAP, VoxPoser, Single Agent) across all 9 RLBench tasks, achieving substantial improvements. For instance, MALMM achieved an average success rate of 0.81, significantly higher than the best baseline at 0.70.
  • Long-Horizon Success: The framework showed particular strength in long-horizon tasks like "stack blocks" and "empty container," where it significantly outperformed the Single Agent approach (e.g., over 6x better success rate for stacking 4 blocks).
  • Real-World Validation: Experiments with a Franka robot arm in real-world settings confirmed MALMM's effectiveness, showing significant improvements over the Single Agent baseline on tasks like "put case" and "rubbish in bin" (40% higher success rate).
  • Robustness: MALMM demonstrated effective failure recovery, as illustrated by its ability to resume stacking after dropping a block, while a single agent hallucinated and continued stacking on the dropped block.

Limitations and Future Work

  • Cost of Operation: The reliance on powerful LLMs like GPT-4-turbo makes the current framework costly to operate. While open-source LLMs can be used, they come with a performance trade-off.
  • Prompt Engineering: The framework still depends on manual prompt engineering, which can be labor-intensive. Future work could explore advancements in prompting techniques to reduce this effort.
  • Object Interaction Complexity: While MALMM handles complex tasks, it still faces limitations with highly intricate object interactions.
  • Vision Pipeline: The current vision pipeline uses single-view observations, which can be susceptible to occlusions. Future work aims to improve this by exploring multi-view settings and vision-language foundation models for richer scene understanding.
  • Future Directions: The authors plan to explore more complex tasks, including those involving articulated objects, and integrate more advanced vision-language models for enhanced contextual scene understanding.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable planning abilities across various domains, including robotic manipulation and navigation. While recent work in robotics deploys LLMs for high-level and low-level planning, existing methods often face challenges with failure recovery and suffer from hallucinations in long-horizon tasks. To address these limitations, we propose a novel multi-agent LLM framework, Multi-Agent Large Language Model for Manipulation (MALMM). Notably, MALMM distributes planning across three specialized LLM agents, namely high-level planning agent, low-level control agent, and a supervisor agent. Moreover, by incorporating environment observations after each step, our framework effectively handles intermediate failures and enables adaptive re-planning. Unlike existing methods, MALMM does not rely on pre-trained skill policies or in-context learning examples and generalizes to unseen tasks. In our experiments, MALMM demonstrates excellent performance in solving previously unseen long-horizon manipulation tasks, and outperforms existing zero-shot LLM-based methods in RLBench by a large margin. Experiments with the Franka robot arm further validate our approach in real-world settings.
BibTeX
@inproceedings{singh2025malmm,
  author = {Harsh Singh and Rocktim Jyoti Das and Mingfei Han and Preslav Nakov and Ivan Laptev},
  title = {{MALMM: Multi-Agent Large Language Models for Zero-Shot Robotic Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11247340},
}
Figure