Introduction
The field of Embodied AI faces a significant evaluation gap. High-level task planning benchmarks often abstract away physical realities, while low-level control benchmarks typically focus on isolated, single-step actions. Real-world robotics requires a seamless integration of both: the cognitive ability to decompose complex natural language instructions and the motor skills to execute them in physically accurate environments. To address this, we introduce Kitchen-R, a novel Isaac Sim-based benchmark designed to evaluate vision-language models (VLMs) and low-level control policies both independently and as a unified system.
The Kitchen-R Framework
Kitchen-R is built upon NVIDIA’s Isaac Sim, providing a photorealistic and physically accurate digital twin of a kitchen environment. The framework is modular, consisting of three primary components:
- Task Planning: A VLM processes natural language instructions and a top-down scene image to generate a sequence of atomic tasks (e.g., "move to table," "pick up juice box").
- Navigation & Manipulation: High-level plans are executed via a path planner (Theta*) for base movement and Riemannian Motion Policies (RMPs) for the ALOHA mobile manipulator's arm, ensuring collision-free trajectories.
- Monitoring & Logging: A real-time system tracks execution, employing a "fail-fast" mechanism that aborts episodes upon failure (e.g., dropping an object), significantly accelerating data collection.
Dataset and Benchmarking
The Kitchen-R dataset includes 563 natural language instructions derived from 58 templates, with task lengths ranging from 4 to 8 steps. Unlike existing suites like OmniGibson or VirtualHome, Kitchen-R provides a complete pipeline for trajectory collection, multimodal sensor support (RGB-D, LiDAR), and ROS integration out of the box.
Researchers can evaluate models using a composite metric that combines Exact Match (EM) for text-based plan accuracy and Success Rate (SR) for physical execution. This modularity allows for "ground-truth" plan injection to isolate whether a failure stemmed from the cognitive planner or the motor policy.
Key Results and Insights
Baseline evaluations using the OmniFusion VLM and Diffusion Policy control models revealed critical insights:
- Prompt Engineering is Essential: Standard VLMs initially failed to generate valid plans (EM: 0.000). However, with in-context examples and constrained generation, performance improved to an EM of 0.632.
- Efficiency in Simulation: The framework enabled the collection of approximately 2,700 mobile manipulation trajectories for the 2024 AIJ Contest.
- Computational Costs: While oracle controllers are fast (~1 min/episode), neural policy inference can extend evaluation to 50 minutes per episode on consumer-grade hardware (RTX 3060 Ti).
Conclusion and Future Work
Kitchen-R bridges the "mind-motion" gap by providing a unified environment for end-to-end robotic evaluation. While currently limited to a single kitchen scene, it establishes a robust foundation for future research. Future iterations will focus on expanding scene diversity, optimizing inference pipelines to reduce evaluation time, and enhancing baseline model robustness for higher end-to-end success rates.
Summary updated