MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation

Harsh Singh, Rocktim Jyoti Das, Mingfei Han, Preslav Nakov, Ivan Laptev

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in high-level reasoning, yet their application to physical robotics remains challenging. Current LLM-based systems often struggle with "hallucinations" during long-horizon tasks—losing track of goals or ignoring physical constraints—and typically operate in an "open-loop" fashion that cannot recover from environmental errors. MALMM (Multi-Agent Large Language Model for Manipulation) addresses these issues by distributing cognitive load across a specialized "Board of Directors" of agents, enabling robust, zero-shot robotic control.

The MALMM Framework: A Specialized Multi-Agent Architecture

The core of MALMM is a multi-agent system that divides complex tasks into manageable roles:

  • The Planner: Handles high-level reasoning, decomposing natural language instructions into logical steps.
  • The Coder: Translates individual steps into executable Python code using robot primitives (e.g., execute_trajectory, close_gripper).
  • The Supervisor: Monitors execution, manages agent transitions, and triggers re-planning upon detecting failures.
A sequence of robotic manipulation steps successfully executed by the MALMM framework.

Unlike traditional systems, MALMM is a closed-loop system. It incorporates real-time environmental feedback—via state-space data or 3D point clouds—after every action. This allows the agents to perceive the results of their commands and adjust dynamically if the environment changes unexpectedly.

Experimental Evaluation and Results

The framework was evaluated on the RLBench benchmark across 9 diverse tasks and on a physical Franka Emika Panda robot arm.

  • Performance: MALMM achieved an 81% success rate on RLBench, significantly outperforming single-agent baselines (50%) and specialized methods like VoxPoser (17%).
  • Long-Horizon Robustness: In complex stacking tasks, MALMM maintained a 28% success rate for 4-block stacks, whereas single-agent performance crashed to a mere 4%.
  • Model Agnostic: While GPT-4-Turbo yielded the best results, MALMM remained effective using open-source models like LLaMA-3.3-70B, which still outperformed GPT-4-based single-agent systems.

Key Insights: The Power of Supervision

A critical finding of the research is that role specialization significantly mitigates hallucinations. By narrowing the context for the Coder, the system avoids the spatial confusion common in monolithic LLM controllers. Furthermore, the Supervisor acts as a vital "sanity check."

Comparison of failure recovery: A single-agent system fails to recognize a dropped block and attempts to stack on empty space (top), while MALMM's closed-loop feedback allows it to recover and complete the task (bottom).

As shown in qualitative analysis, when a robot drops an object, a single-agent system often continues the plan as if the object were still in place. MALMM’s Supervisor identifies the discrepancy between the expected and actual state, prompting the Planner to re-sequence the task and pick the object back up before proceeding.

Limitations and Future Outlook

Despite its success, MALMM faces challenges regarding latency due to multiple LLM calls per step, making it less suitable for high-speed applications. Additionally, its reliance on 3D bounding boxes for perception can be a bottleneck in cluttered environments or with complex, non-rigid geometries. Future work aims to integrate vision-language foundation models to provide richer contextual scene details and improve real-time performance.

Summary updated

BibTeX
@inproceedings{singh2025malmm,
  author = {Harsh Singh and Rocktim Jyoti Das and Mingfei Han and Preslav Nakov and Ivan Laptev},
  title = {{MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11247340},
}
Figure