Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation

Nikita Kachaev, Andrei Spiridonov, Andrey S. Gorodetsky, Kirill Muravyev, Nikita Oskolkov, Aditya Narendra, Vlad Shakhuro, Dmitry Makarov, Aleksandr I. Panov, Polina Fedotova, Alexey K. KovalevarXiv 2025 · DOI · arXiv

Introduction

The field of Embodied AI faces a significant evaluation gap. High-level task planning benchmarks often abstract away physical realities, while low-level control benchmarks typically focus on isolated, single-step actions. Real-world robotics requires a seamless integration of both: the cognitive ability to decompose complex natural language instructions and the motor skills to execute them in physically accurate environments. To address this, we introduce Kitchen-R, a novel Isaac Sim-based benchmark designed to evaluate vision-language models (VLMs) and low-level control policies both independently and as a unified system.

The Kitchen-R Framework

Kitchen-R is built upon NVIDIA’s Isaac Sim, providing a photorealistic and physically accurate digital twin of a kitchen environment. The framework is modular, consisting of three primary components:

  1. Task Planning: A VLM processes natural language instructions and a top-down scene image to generate a sequence of atomic tasks (e.g., "move to table," "pick up juice box").
  2. Navigation & Manipulation: High-level plans are executed via a path planner (Theta*) for base movement and Riemannian Motion Policies (RMPs) for the ALOHA mobile manipulator's arm, ensuring collision-free trajectories.
  3. Monitoring & Logging: A real-time system tracks execution, employing a "fail-fast" mechanism that aborts episodes upon failure (e.g., dropping an object), significantly accelerating data collection.

Dataset and Benchmarking

The Kitchen-R dataset includes 563 natural language instructions derived from 58 templates, with task lengths ranging from 4 to 8 steps. Unlike existing suites like OmniGibson or VirtualHome, Kitchen-R provides a complete pipeline for trajectory collection, multimodal sensor support (RGB-D, LiDAR), and ROS integration out of the box.

Researchers can evaluate models using a composite metric that combines Exact Match (EM) for text-based plan accuracy and Success Rate (SR) for physical execution. This modularity allows for "ground-truth" plan injection to isolate whether a failure stemmed from the cognitive planner or the motor policy.

Key Results and Insights

Baseline evaluations using the OmniFusion VLM and Diffusion Policy control models revealed critical insights:

  • Prompt Engineering is Essential: Standard VLMs initially failed to generate valid plans (EM: 0.000). However, with in-context examples and constrained generation, performance improved to an EM of 0.632.
  • Efficiency in Simulation: The framework enabled the collection of approximately 2,700 mobile manipulation trajectories for the 2024 AIJ Contest.
  • Computational Costs: While oracle controllers are fast (~1 min/episode), neural policy inference can extend evaluation to 50 minutes per episode on consumer-grade hardware (RTX 3060 Ti).

Conclusion and Future Work

Kitchen-R bridges the "mind-motion" gap by providing a unified environment for end-to-end robotic evaluation. While currently limited to a single kitchen scene, it establishes a robust foundation for future research. Future iterations will focus on expanding scene diversity, optimizing inference pipelines to reduce evaluation time, and enhancing baseline model robustness for higher end-to-end success rates.

Summary updated

BibTeX
@misc{kachaev2025mind,
  author = {Nikita Kachaev and Andrei Spiridonov and Andrey S. Gorodetsky and Kirill Muravyev and Nikita Oskolkov and Aditya Narendra and Vlad Shakhuro and Dmitry Makarov and Aleksandr I. Panov and Polina Fedotova and Alexey K. Kovalev},
  title = {{Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation}},
  year = {2025},
  eprint = {2508.15663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.15663},
}
Figure