MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models

Kanxue Li, Baosheng Yu, Qi Zheng, Yibing Zhan, Yuhui Zhang, Tianle Zhang, Yijun Yang, Yue Chen, Lei Sun, Qiong Cao, Li Shen, Lusong Li, Dapeng Tao, Xiaodong He

Abstract: Foundation models have demonstrated significant emergent abilities, holding great promise for enhancing embodied agents' reasoning and planning capacities. However, the absence of a comprehensive benchmark for evaluating embodied agents with multimodal observations in complex environments remains a notable gap. In this paper, we present MuEP, a comprehensive Multimodal benchmark for Embodied Planning. MuEP facilitates the evaluation of multimodal and multi-turn interactions of embodied agents in complex scenes, incorporating fine-grained evaluation metrics that provide insights into the performance of embodied agents throughout each task. Furthermore, we evaluate embodied agents with recent state-of-the-art foundation models, including large language models (LLMs) and large multimodal models (LMMs), on the proposed benchmark. Experimental results show that foundation models based on textual representations of environments usually outperform their visual counterparts, suggesting a gap in embodied planning abilities with multimodal observations. We also find that control language generation is an indispensable ability beyond common-sense knowledge for accurate embodied task completion. We hope the proposed MuEP benchmark can contribute to the advancement of embodied AI with foundation models.

Introduction

For an AI agent to be truly "embodied"—functioning like a robot in a domestic setting—it must perceive its environment, reason about complex goals (e.g., "heat up a potato"), and execute a sequence of multi-turn actions. While Large Language Models (LLMs) have shown exceptional reasoning capabilities, applying them to physical tasks remains a significant challenge.

Existing benchmarks often suffer from limited scale, focus on a single modality (text or video only), or rely on "coarse" metrics like simple success/failure rates. These metrics fail to explain why an agent failed—whether it was due to object hallucination, failure to follow command formats, or logical loops. MuEP addresses these gaps by providing a massive, multimodally-aligned dataset and fine-grained metrics to pinpoint specific cognitive failures.

Methodology: The MuEP Pipeline

The MuEP benchmark is constructed through an automated four-step pipeline designed to ensure high-quality, diverse, and physically grounded tasks:

  1. Metadata Collection: Environmental data is extracted from the Ai2-THOR simulator. Using PDDL (Planning Domain Definition Language), the system ensures perfect alignment between visual scenes and textual descriptions.
  2. Task Generation: GPT-3.5 is utilized to brainstorm 30 diverse tasks per scene, ranging from simple placement to complex multi-object interactions.
  3. Heuristic Filtering: A rigorous filtering process removes impossible tasks, such as attempting to use a microwave in a room where none exists.
  4. Multimodal Demonstration: A rule-based planner generates "expert" trajectories, resulting in a dataset of 176,593 image-text pairs.
The MuEP pipeline: from metadata collection and task generation to multimodal demonstration construction.

Fine-Grained Evaluation Metrics

Beyond the standard Success Rate (SR), MuEP introduces two novel metrics to better understand agent behavior:

  • Language Compliance (LC): This measures the model's ability to output actions in the specific structured format required by the environment (e.g., "take mug 1 from table 1" vs. a generic "pick up mug").
  • Reasoning Disorientation Index (RDI): This quantifies how often an agent gets "stuck" in a loop, repeating the same ineffective action despite receiving negative feedback from the environment.

Experimental Results and Analysis

The authors evaluated eight state-of-the-art models, including text-only LLMs (LLaMA-2, OPT, Bloomz) and Large Multimodal Models (LMMs) such as BLIP2, LLaVA, and MiniGPT-4.

The Modality Gap

A primary finding is the significant "vision gap." Models using textual representations of the environment consistently outperformed those using visual observations. For instance, the text-only OPT-6.7B model achieved a Success Rate of 79.10% on unseen tasks, whereas the best-performing multimodal model, MiniGPT-4, reached only 25.37%. This suggests that current LMMs struggle to translate raw pixels into the high-level logical "state" that LLMs receive directly in text-based formats.

Radar charts comparing model performance across SR, IS, GCS, LC, and RDI for both seen and unseen tasks.

Bottlenecks in Planning

  • Format Compliance: A major failure mode was not a lack of common sense, but a failure in Language Compliance. If a model cannot strictly format its output, the system fails to execute the command.
  • Cognitive Traps: The RDI metric revealed that agents often lack the self-correction needed for embodied tasks. When an action fails, they frequently repeat the mistake rather than replanning, highlighting a deficiency in historical context integration.
  • Task Complexity: "Pick two & place" tasks proved the most difficult, with some models scoring 0% success due to the complexity of maintaining state over multiple steps.

Conclusion

MuEP reveals that while foundation models hold promise, a substantial gap remains in their ability to plan using multimodal observations. Future research must focus on closing this modality gap, improving language compliance for control, and developing architectures that better utilize interaction history to avoid logical disorientation. MuEP provides the necessary infrastructure to track progress in these critical areas of embodied AI.

Summary updated

Abstract

Foundation models have demonstrated significant emergent abilities, holding great promise for enhancing embodied agents' reasoning and planning capacities. However, the absence of a comprehensive benchmark for evaluating embodied agents with multimodal observations in complex environments remains a notable gap. In this paper, we present MuEP, a comprehensive Multimodal benchmark for Embodied Planning. MuEP facilitates the evaluation of multimodal and multi-turn interactions of embodied agents in complex scenes, incorporating fine-grained evaluation metrics that provide insights into the performance of embodied agents throughout each task. Furthermore, we evaluate embodied agents with recent state-of-the-art foundation models, including large language models (LLMs) and large multimodal models (LMMs), on the proposed benchmark. Experimental results show that foundation models based on textual representations of environments usually outperform their visual counterparts, suggesting a gap in embodied planning abilities with multimodal observations. We also find that control language generation is an indispensable ability beyond common-sense knowledge for accurate embodied task completion. We hope the proposed MuEP benchmark can contribute to the advancement of embodied AI with foundation models.
BibTeX
@inproceedings{li2024muep,
  author = {Kanxue Li and Baosheng Yu and Qi Zheng and Yibing Zhan and Yuhui Zhang and Tianle Zhang and Yijun Yang and Yue Chen and Lei Sun and Qiong Cao and Li Shen and Lusong Li and Dapeng Tao and Xiaodong He},
  title = {{MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=PHqhe0fKUz},
}
Figure