ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments

Taewoong Kim, Cheolhong Min, Byeonghwi Kim, Jinyeon Kim, Wonje Jeung, Jonghyun Choi

TL;DR

This paper introduces ReALFRED, a new benchmark for embodied AI that bridges the gap between simulated and real-world environments. By using 3D-captured, multi-room scenes with interactive objects and free-form language instructions, ReALFRED aims to train robotic agents that can perform complex household tasks more effectively and be more readily deployed in real-world scenarios.

What Problem Does This Paper Address?

Training robotic agents to perform household tasks in simulated environments has seen significant progress. However, these simulations often suffer from limited object interactivity, a visual appearance that differs from the real world, and smaller environment sizes. These limitations hinder the direct deployment of models trained in simulation to real-world robots. The authors aim to address this "simulation-to-reality" (sim2real) gap by creating a benchmark that offers more realistic environments, specifically by incorporating 3D-captured scenes, larger multi-room layouts, and a wider range of interactive objects.

Methods

The core contribution is the ReALFRED benchmark. This benchmark extends the existing ALFRED benchmark by:

  • 3D-Captured Environments: Utilizing real-world scanned indoor scenes to reduce the visual domain gap.
  • Multi-Room Layouts: Environments consist of multiple rooms, mimicking real-world house layouts and requiring agents to navigate across them.
  • Interactive Objects: Objects within the scenes are made interactive, allowing for more complex task execution beyond simple navigation.
  • Free-Form Language Instructions: The benchmark uses natural, unconstrained language instructions for tasks, demanding better language understanding from agents.
  • Expert Demonstrations: The dataset includes expert demonstrations of task completion, providing valuable training data.

The authors also conducted experiments to evaluate existing state-of-the-art methods on their new benchmark, demonstrating that models trained on less realistic environments struggle to perform well on ReALFRED. They also explored sim-to-real domain adaptation techniques.

Datasets

  • ReALFRED Benchmark: The primary contribution is this newly introduced benchmark. It comprises 150 3D-captured indoor houses, featuring interactive objects and free-form language instructions for household tasks.
  • ALFRED Benchmark: ReALFRED builds upon and extends the ALFRED benchmark, incorporating its task structure and instruction format but with more realistic environments.
  • Existing Datasets (for comparison): The paper references and compares ReALFRED to other benchmarks like Matterport3D, HM3D, Gibson, AI2-THOR, and RoboTHOR.

Key Insights and Ideas

  • The "Reality Gap" is Significant: Existing state-of-the-art methods, which perform well on simulated benchmarks like ALFRED, show a substantial drop in performance on ReALFRED. This highlights the critical need for more realistic training environments.
  • Multi-Room Navigation is a Major Hurdle: The larger, multi-room nature of ReALFRED scenes poses a significant challenge for agents, requiring more complex navigation strategies and better understanding of spatial relationships between rooms.
  • Object Interaction is Crucial for Real-World Tasks: The inclusion of interactive objects is essential for training agents that can perform actual household chores, moving beyond mere navigation.
  • Sim2Real Adaptation Still Faces Challenges: While domain adaptation techniques show some promise, a significant performance gap remains between agents trained in simulation and those trained on real data, underscoring the difficulty of bridging the visual domain gap.

Key Results

  • Lower Performance for Existing Methods: State-of-the-art models evaluated on ReALFRED consistently achieved lower success rates (SR) and goal-condition success rates (GC) compared to their performance on ALFRED. For instance, the best-performing "Imitation Learning" baseline achieved an SR of 13.45% on ReALFRED's seen validation split, compared to much higher figures reported for ALFRED.
  • Larger Environments Increase Difficulty: The paper shows that agents perform significantly worse in larger ReALFRED scenes. The average SR for the best model dropped from 5.46% in smaller scenes to 1.77% in larger scenes.
  • Navigation Success Rate: The navigation success rate in ReALFRED was found to be 59.18% for the valid unseen split, considerably lower than the 84.82% reported for ALFRED.
  • Sim2Real Performance: A sim-to-real agent with domain adaptation (using CycleGan or UVCGAN-v2) still underperformed a real-to-real agent, indicating the ongoing challenges in sim2real transfer.

Limitations and Future Work

  • Task Diversity: While ReALFRED offers more object classes and unique episodes, the types of tasks are still somewhat limited, especially when considering more complex real-world scenarios that might require two-handed manipulation.
  • Language Support: The current benchmark only supports natural language instructions in English.
  • Future Work Suggestions:
    • Adding more complicated tasks that require two-handed manipulation.
  • Supporting multi-lingual interfaces to cater to users from different regions.

Summary updated

Abstract

Simulated virtual environments have been widely used to learn robotic agents that perform daily household tasks. These environments encourage research progress by far, but often provide limited object interactability, visual appearance different from real-world environments, or relatively smaller environment sizes. This prevents the learned models in the virtual scenes from being readily deployable. To bridge the gap between these learning environments and deploying (i.e., real) environments, we propose the ReALFRED benchmark that employs real-world scenes, objects, and room layouts to learn agents to complete household tasks by understanding free-form language instructions and interacting with objects in large, multi-room and 3D-captured scenes. Specifically, we extend the ALFRED benchmark with updates for larger environmental spaces with smaller visual domain gaps. With ReALFRED, we analyze previously crafted methods for the ALFRED benchmark and observe that they consistently yield lower performance in all metrics, encouraging the community to develop methods in more realistic environments. Our code and data are publicly available (Homepage: https://github.com/snumprlab/realfred).
BibTeX
@inproceedings{kim2024realfred,
  author = {Taewoong Kim and Cheolhong Min and Byeonghwi Kim and Jinyeon Kim and Wonje Jeung and Jonghyun Choi},
  title = {{ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72684-2\_20},
}
Figure