OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following

Haochen Shi, Zhiyuan Sun, Xingdi Yuan, Marc-Alexandre Côté, Bang Liu

Abstract

Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components—ranging from visual perception to action execution—on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by deploying a multi-agent dialogue strategy on a TextWorld counterpart, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.

Introduction

Embodied Instruction Following (EIF) represents a significant challenge in personal robotics, requiring an agent to interpret complex natural language commands (e.g., "wash a dirty bowl and put it in the cupboard") and translate them into a series of physical interactions within a dynamic environment. While Large Language Models (LLMs) have been integrated into these systems to handle high-level reasoning, the field has lacked a systematic understanding of which specific components—perception, planning, or execution—drive success or cause failure. The OPEx framework was developed to isolate these variables and identify the primary bottlenecks in current state-of-the-art systems.

The OPEx Framework

The OPEx framework architecture, illustrating the interaction between the Observer, Planner, and Executor modules.

The OPEx architecture decouples the cognitive and physical requirements of an embodied agent into three specialized modules:

  1. The Observer: This LLM-based module (utilizing GPT-3.5-turbo) processes raw environmental data, such as detected objects and error messages, into concise, task-centric text descriptions. This acts as an information filter, preventing the agent from being overwhelmed by irrelevant visual noise.
  2. The Planner: A high-level reasoning module (GPT-4) that employs Chain-of-Thought (CoT) prompting to decompose complex instructions into a sequence of logical subtasks.
  3. The Executor: Inspired by the ReAct framework, this module (GPT-4) receives the current subtask and the Observer’s summary to select specific "skills" (e.g., NavigateTo, Explore, PickUp) from a pre-defined library.

To support long-term memory, the system incorporates a Semantic Mapping Module that constructs a 2D top-down map of the environment from egocentric RGB-D images, allowing the agent to track objects across different rooms.

Experimental Results

OPEx was evaluated on two primary benchmarks: ALFRED, which focuses on long-horizon household tasks requiring visual perception, and ALFWorld, a text-based counterpart used to isolate pure reasoning capabilities.

Performance comparison of OPEx against baseline methods on the ALFRED benchmark.

As shown in the results, OPEx-S achieved a 41.27% success rate (SR) on the ALFRED "Unseen" test split, significantly outperforming the previous modular state-of-the-art, FILM (24.46%). Furthermore, OPEx demonstrated a robust 53.82% Goal Condition (GC) completion rate in unseen environments, indicating its ability to make substantial progress even on tasks it did not fully complete.

Key Insights and Analysis

  • The Perception Bottleneck: Ablation studies revealed that when the agent is provided with "perfect" ground-truth perception, its success rate nearly doubles (increasing from approximately 36% to 59%). This suggests that current LLM reasoning capabilities far exceed the accuracy of existing computer vision models.
  • Data Efficiency: Unlike traditional models requiring thousands of expert demonstrations, the LLM-based Planner achieved superior results using less than 10% of the training data, leveraging the massive common-sense priors inherent in LLMs.
  • Multi-Agent Specialization: On the ALFWorld benchmark, splitting the agent into specialized "Reasoner" and "Actor" roles that communicate via dialogue significantly boosted performance, preventing the model from losing track of high-level plans while managing low-level navigation.
  • Execution Gaps: Error analysis identified "Goal object not found" (~27%) as the most common failure mode. This highlights a critical dependency: if the vision system fails to detect an object during initial exploration, the reasoning system cannot recover.

Conclusion

OPEx demonstrates that LLM-centric designs provide a powerful foundation for embodied agents, particularly in terms of reasoning and data efficiency. However, the study clearly identifies visual perception and low-level action execution as the primary remaining bottlenecks. Future research should focus on tighter vision-language integration, potentially through native Vision-Language Models (VLMs), and reducing the latency and cost associated with high-end proprietary LLMs.

Summary updated

Abstract

Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components-ranging from visual perception to action execution-on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by deploying a multi-agent dialogue strategy on a TextWorld counterpart, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.
BibTeX
@inproceedings{shi2024opex,
  author = {Haochen Shi and Zhiyuan Sun and Xingdi Yuan and Marc-Alexandre Côté and Bang Liu},
  title = {{OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ekpM7kuoDG},
}
Figure