Abstract
Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components—ranging from visual perception to action execution—on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by integrating a multi-agent design into the Planner component of our LLM-centric architecture, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.
Introduction
Embodied Instruction Following (EIF) represents a significant challenge in AI, requiring agents to interpret natural language commands, navigate 3D environments, and manipulate objects based on egocentric visual observations. While Large Language Models (LLMs) have recently been integrated into EIF frameworks to serve as high-level "brains," a unified understanding of how individual components—from perception to execution—contribute to overall success has been lacking. This paper introduces OPEx, a framework designed to provide a component-wise analysis of LLM-centric agents, identifying critical bottlenecks and proposing architectural improvements.
The OPEx Framework
The OPEx framework decomposes the agent into three specialized roles to streamline the transition from raw sensory input to physical action:
- Observer: This component processes RGB images and semantic maps to generate natural language descriptions of the environment. By acting as an information filter, it provides the LLM with relevant context while discarding visual noise.
- Planner: Utilizing an LLM (such as GPT-4) with Chain-of-Thought reasoning, the Planner breaks down high-level goal instructions into a sequence of manageable subtasks.
- Executor: The Executor receives the current subtask and the Observer's environmental description. It selects specific skills (e.g.,
NavigateTo,PickupObject) from a library using a ReAct (Reason + Act) paradigm to handle environmental feedback and errors.
Experimental Analysis and Results
The researchers evaluated OPEx on the ALFRED and ALFWorld benchmarks. ALFRED provides a complex 3D simulation for household tasks, while ALFWorld serves as a text-based counterpart to isolate reasoning capabilities from visual and physical challenges.
Performance and Data Efficiency
OPEx demonstrated state-of-the-art performance, achieving a 41.27% Success Rate (SR) on the ALFRED "Test Unseen" split, a significant jump over previous baselines like FILM (24.46%). Notably, OPEx achieved these results with extreme data efficiency, requiring less than 10% of the training data used by traditional supervised models by leveraging the in-context learning capabilities of LLMs.
Identifying Bottlenecks
A core contribution of this work is the systematic analysis of component influence. By providing the agent with "perfect" ground-truth perception (OPEx-P), the success rate on unseen environments increased from approximately 41% to nearly 60%. This indicates that while LLM reasoning is highly capable, current computer vision models (the "Observer") remain a primary bottleneck.
Key Insights
- Skill Augmentation: The introduction of a
LookAroundskill significantly improved planning quality by allowing the agent to build a more accurate mental map of the environment before committing to subtasks. - Multi-Agent Synergy: In text-based environments (ALFWorld), a multi-agent design—splitting responsibilities between a "Reasoner" and an "Actor"—helped mitigate hallucinations and improved grounding.
Conclusion
The OPEx framework highlights that while LLMs excel at high-level planning for EIF, the path to fully autonomous household robots is currently constrained by visual perception and low-level execution. Future research should focus on reducing the inference overhead of large models and developing more robust vision-to-language modules capable of handling complex spatial relationships and small objects.
Summary updated