LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments

Zixia Jia, Mengmeng Wang, Baichen Tong, Song-Chun Zhu, Zilong Zheng

Abstract

Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents' capacity to develop "internalized world knowledge" with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuitE represents a significant step toward building embodied generalists in the context of language models.

Introduction

While Large Language Models (LLMs) like GPT-4 demonstrate sophisticated reasoning, their performance as "embodied agents"—entities that perceive and act within dynamic environments—remains a significant challenge. Most existing benchmarks treat LLMs as high-level planners receiving a "god-view" or global observation of a scene. In reality, an agent's perception is limited to its immediate surroundings. Furthermore, traditional embodied benchmarks often require heavy 3D simulation engines, which are computationally expensive and difficult to customize.

LangSuitE addresses these issues by providing a lightweight, simulation-free framework to test if LLMs can build an "internalized world model" from partial, textual observations alone. By formulating embodied tasks as a Partially Observable Markov Decision Process (POMDP), LangSuitE isolates the reasoning and planning capabilities of the model from visual perception errors.

The LangSuitE Testbed

LangSuitE utilizes natural language for both observations and actions. An agent receives a text description of its immediate environment (e.g., "On your left, you see a dining table") and must output discrete, low-level actions (e.g., move_ahead, turn_left). This text-based approach is approximately 4x faster than AI2Thor and 35x faster than OmniGibson, facilitating rapid experimentation.

Overview of the LangSuitE environment and its six representative embodied tasks, including Interactive QA and Multi-agent Cooperation.

Embodied Observations vs. Global Knowledge

A key distinction of LangSuitE compared to previous benchmarks like ALFWorld is the shift from global to embodied descriptions. In ALFWorld, agents often receive a full inventory of the room at the start. In LangSuitE, the agent must navigate to discover objects, mimicking real-world constraints. Similarly, while ALFWorld uses high-level commands like goto the cabinet, LangSuitE requires low-level navigation commands, forcing the agent to manage the fine-grained execution of its plans.

Comparison between ALFWorld's global descriptions/high-level actions and LangSuitE's embodied descriptions/low-level actions.

EmMem: Addressing the Memory Bottleneck

The researchers identified that standard "Chain-of-Thought" (CoT) and "Reason + Act" (ReAct) strategies often fail in long-horizon tasks due to the "lost-in-the-middle" effect, where models forget earlier observations or hallucinate states. To combat this, the authors introduced EmMem (Embodied Memory).

EmMem is a memory-enhanced prompting strategy that requires the model to explicitly summarize its current state—including its location, history, and active sub-goals—before predicting the next action. This acts as a mental map, helping the agent maintain self-awareness and consistency across hundreds of interaction turns.

Benchmarking and Results

LangSuitE integrates six representative tasks adapted from embodied AI literature:

  1. Instruction Grounding (IG): Navigating to objects based on specific commands.
  2. Rearrangement: Returning misplaced items to their original locations.
  3. Interactive QA (IQA): Exploring a room to answer questions about its contents.
  4. Household: Executing complex, multi-step chores (e.g., cleaning a mug).
  5. Multi-agent Teach: A "Commander" agent guiding a "Follower."
  6. Multi-agent Cooperation: Two agents collaborating to finish a shared task.

Benchmarking results using gpt-3.5-turbo highlight the difficulty of these tasks. In the complex "Household" task using low-level actions, the Reflexion + EmMem strategy achieved a success rate of 14.0%, nearly doubling baseline performance. In Interactive QA, EmMem boosted accuracy from 14.9% to 60.0%, demonstrating that memory is vital for effective exploration. Multi-agent cooperative tasks showed higher success rates (64.0%), indicating that LLMs are proficient at dividing labor through text-based negotiation.

Conclusion

LangSuitE provides a rigorous testing ground for the next generation of embodied generalists. While the current results show a significant gap between high-level planning and low-level execution, the introduction of EmMem demonstrates that structured memory can significantly mitigate the planning failures of LLMs. Future work will look toward bridging this textual reasoning with Vision-Language Models (VLMs) to move closer to real-world robotic applications.

Summary updated

Abstract

Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents' capacity to develop ``internalized world knowledge'' with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuitE represents a significant step toward building embodied generalists in the context of language models.
BibTeX
@inproceedings{jia2024langsuite,
  author = {Zixia Jia and Mengmeng Wang and Baichen Tong and Song-Chun Zhu and Zilong Zheng},
  title = {{LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TCP6haLRjT},
}
Figure