MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models

Kanxue Li, Baosheng Yu, Qi Zheng

Introduction

Embodied AI—the field of creating agents that can perceive and act within an environment—has recently pivoted toward using Large Language Models (LLMs) and Large Multimodal Models (LMMs) as "brains." While these foundation models possess immense general knowledge, their ability to function as effective planners in complex, stochastic 3D environments remains a significant research challenge. Evaluating these agents is notoriously difficult; existing benchmarks often suffer from limited scale due to the high cost of manual annotation, a focus on single modalities (either text or vision), and coarse metrics like "Success Rate" that fail to diagnose specific failure modes.

To address these gaps, researchers have introduced MuEP (Multimodal Embodied Planning), a massive new multimodal benchmark designed to test how well foundation models can plan and execute complex tasks. By providing a diverse, large-scale dataset with fine-grained metrics, MuEP allows for a deeper understanding of where today’s models succeed and where they stumble.

The MuEP Pipeline

The core of this work is an automated pipeline designed to build the MuEP benchmark through four distinct stages. This approach ensures a high degree of task diversity and scale without the bottlenecks associated with human-in-the-loop data collection.

The four-stage automated pipeline for MuEP benchmark construction, integrating metadata collection, task generation, filtering, and multimodal demonstration generation.
  1. Metadata Collection: The process begins by extracting "ground truth" metadata—including object locations, states, and attributes—from the Ai2-THOR 3D simulator. This data is then aligned with precise textual descriptions using the TextWorld engine, creating a unified representation of the environment.
  2. Task Generation: Leveraging the reasoning capabilities of GPT-3.5, the pipeline generates 30 diverse tasks for each scene through in-context learning. Tasks range from simple "pick and place" to complex multi-step instructions like "put a heated potato on the table."
  3. Task Filtering: To ensure quality, a heuristic filter removes impossible or redundant tasks, such as instructions to place items in objects that are not containers.
  4. Multimodal Demonstration: Finally, a rule-based PDDL (Planning Domain Definition Language) planner is used to generate "expert" paths. This results in a comprehensive dataset of paired images and text-based actions that serve as the gold standard for agent training and evaluation.

Dataset Characteristics

The resulting MuEP dataset represents a significant leap in scale and complexity. It includes 15,247 expert demonstration episodes and 176,593 image-text pairs across 108 varied household scenes. With a wide array of objects, textures, and layouts, MuEP is uniquely positioned to support the evaluation of both text-only (LLM) and vision-language (LMM) agents, providing a more robust testing ground than previous benchmarks like TaPA or ALFRED.

Evaluation and the "Modality Gap"

The researchers evaluated eight open-source foundation models, including LLaMA-2, OPT, BLIP2, and LLaVA, using Parameter-Efficient Fine-Tuning (PEFT) via Q-LoRA. The evaluation went beyond simple success rates, introducing fine-grained metrics:

  • Language Compliance (LC): Measures how well an agent follows the specific command format required by the simulator.
  • Reasoning Disorientation Index (RDI): Tracks "cognitive traps" where an agent repeats the same failed action indefinitely.
Radar charts comparing the performance of eight foundation models across five metrics in both seen and unseen environments.

A surprising "modality gap" emerged: despite having access to visual data, multimodal models (LMMs) were consistently outperformed by text-only models (LLMs) that received textual descriptions of the scene. For example, OPT-6.7B achieved a Success Rate (SR) of 79.10% on unseen tasks, the highest in the study. While MiniGPT-4 was the strongest performer among LMMs, it still lagged behind the top LLMs. This suggests that current LMMs struggle to efficiently extract the logical "state" of a room from raw pixels to form a coherent plan.

Key Insights and Failure Modes

The study pinpointed several reasons for agent failure. A major hurdle was Language Compliance; for complex tasks like "Pick two & place," models like LLaMA-2-7B saw a 0% success rate simply because they could not consistently output commands in the required format. Furthermore, agents often suffered from Reasoning Disorientation, failing to process environmental feedback and getting stuck in repetitive loops. Finally, while models showed competence in familiar rooms, their efficiency (measured by Interaction Steps) plummeted in "unseen" environments, highlighting a significant challenge in generalization.

Conclusion

MuEP provides a critical diagnostic tool for the embodied AI community. The findings underscore a persistent gap between visual perception and logical planning in current foundation models. Future research will focus on scaling scene complexity and developing new architectures that help multimodal models bridge the gap between "seeing" and "doing."

Summary updated

BibTeX
@inproceedings{li2024muep,
  author = {Kanxue Li and Baosheng Yu and Qi Zheng},
  title = {{MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=PHqhe0fKUz},
}
Figure