Abstract
Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components—ranging from visual perception to action execution—on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by deploying a multi-agent dialogue strategy on a TextWorld counterpart, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.
Introduction
Embodied Instruction Following (EIF) represents a significant challenge in personal robotics, requiring an agent to interpret complex natural language commands (e.g., "wash a dirty bowl and put it in the cupboard") and translate them into a series of physical interactions within a dynamic environment. While Large Language Models (LLMs) have been integrated into these systems to handle high-level reasoning, the field has lacked a systematic understanding of which specific components—perception, planning, or execution—drive success or cause failure. The OPEx framework was developed to isolate these variables and identify the primary bottlenecks in current state-of-the-art systems.
The OPEx Framework
The OPEx architecture decouples the cognitive and physical requirements of an embodied agent into three specialized modules:
- The Observer: This LLM-based module (utilizing GPT-3.5-turbo) processes raw environmental data, such as detected objects and error messages, into concise, task-centric text descriptions. This acts as an information filter, preventing the agent from being overwhelmed by irrelevant visual noise.
- The Planner: A high-level reasoning module (GPT-4) that employs Chain-of-Thought (CoT) prompting to decompose complex instructions into a sequence of logical subtasks.
- The Executor: Inspired by the ReAct framework, this module (GPT-4) receives the current subtask and the Observer’s summary to select specific "skills" (e.g.,
NavigateTo,Explore,PickUp) from a pre-defined library.
To support long-term memory, the system incorporates a Semantic Mapping Module that constructs a 2D top-down map of the environment from egocentric RGB-D images, allowing the agent to track objects across different rooms.
Experimental Results
OPEx was evaluated on two primary benchmarks: ALFRED, which focuses on long-horizon household tasks requiring visual perception, and ALFWorld, a text-based counterpart used to isolate pure reasoning capabilities.
As shown in the results, OPEx-S achieved a 41.27% success rate (SR) on the ALFRED "Unseen" test split, significantly outperforming the previous modular state-of-the-art, FILM (24.46%). Furthermore, OPEx demonstrated a robust 53.82% Goal Condition (GC) completion rate in unseen environments, indicating its ability to make substantial progress even on tasks it did not fully complete.
Key Insights and Analysis
- The Perception Bottleneck: Ablation studies revealed that when the agent is provided with "perfect" ground-truth perception, its success rate nearly doubles (increasing from approximately 36% to 59%). This suggests that current LLM reasoning capabilities far exceed the accuracy of existing computer vision models.
- Data Efficiency: Unlike traditional models requiring thousands of expert demonstrations, the LLM-based Planner achieved superior results using less than 10% of the training data, leveraging the massive common-sense priors inherent in LLMs.
- Multi-Agent Specialization: On the ALFWorld benchmark, splitting the agent into specialized "Reasoner" and "Actor" roles that communicate via dialogue significantly boosted performance, preventing the model from losing track of high-level plans while managing low-level navigation.
- Execution Gaps: Error analysis identified "Goal object not found" (~27%) as the most common failure mode. This highlights a critical dependency: if the vision system fails to detect an object during initial exploration, the reasoning system cannot recover.
Conclusion
OPEx demonstrates that LLM-centric designs provide a powerful foundation for embodied agents, particularly in terms of reasoning and data efficiency. However, the study clearly identifies visual perception and low-level action execution as the primary remaining bottlenecks. Future research should focus on tighter vision-language integration, potentially through native Vision-Language Models (VLMs), and reducing the latency and cost associated with high-end proprietary LLMs.
Summary updated