PR2: A Physics- and Photo-realistic Testbed for Embodied AI and Humanoid Robots

Hangxin Liu, Qi Xie, Zeyu Zhang, Tao Yuan, Xiaokun Leng, Lining Sun, Song-Chun Zhu, Jingwen Zhang, Zhicheng He, Yao Su

Introduction

Humanoid robotics research has long been hindered by a significant accessibility barrier: the high cost and maintenance requirements of physical hardware. While simulation offers a scalable alternative, the field has historically been divided between two paradigms. The "Robotics Camp" utilizes platforms like MuJoCo or Gazebo, which provide high-fidelity physics but lack the visual realism necessary for modern computer vision. Conversely, the "Embodied AI Camp" uses environments like Habitat or AI2-THOR, which offer photorealistic rendering but rely on kinematic approximations that bypass the complex dynamics of physical interaction.

This paper introduces PR2, a novel simulation testbed designed to bridge this dichotomy. Built on NVIDIA Isaac Sim, PR2 enables the study of dynamic whole-body motions within environments that are both physically accurate and visually indistinguishable from the real world.

Overview of the PR2 testbed, highlighting its dual focus on photorealistic rendering and fine-grained physical simulation across indoor and outdoor environments.

System Architecture and Methodology

PR2 leverages the NVIDIA PhysX 5 engine for GPU-accelerated physics and real-time ray-traced rendering. Its architecture is characterized by three primary innovations:

  1. Co-simulation Architecture: The platform utilizes Universal Scene Description (USD) assets to simultaneously manage rigid-body dynamics and high-quality scene rendering.
  2. Modular Control Framework: To facilitate research for both beginners and experts, PR2 integrates a baseline walking and jumping controller (based on Drake). This modularity allows researchers to focus on high-level AI tasks or swap in custom whole-body controllers.
  3. Multi-modal VLA Interface: The testbed includes specialized APIs for integrating Large Language Models (LLMs) and Vision-Language Models (VLMs), supporting complex Vision-Language-Action (VLA) tasks.
The PR2 system architecture, illustrating the integration of NVIDIA Omniverse, PhysX, and modular controllers for kino-dynamic trajectory planning and multi-modal AI interaction.

Benchmarking Humanoid Capabilities

The authors validated PR2 through a nationwide competition involving 137 teams and 400 participants. The benchmark utilized the KUAVO v1.0 humanoid robot (1.2m tall, 18-DOF) across six progressively difficult tasks:

  • Locomotion: Navigating uneven rocky terrain and climbing 7° slopes.
  • Loco-manipulation: Turning valves (requiring torque application while balancing) and precision button pressing.
  • Dynamics: Executing vertical jumps to test kinodynamic limits.
  • Full-stack AI: Indoor object search and transport, requiring language parsing, navigation, and manipulation.

Key Insights and Results

The large-scale deployment of PR2 revealed several critical bottlenecks in current humanoid research:

  • The Manipulation Gap: While participants performed reasonably well in locomotion (46% average score), performance dropped significantly in manipulation (29%) and perception (19%). This confirms that integrating vision with hand-eye coordination remains a primary challenge.
  • The Compute Trilemma: High-fidelity simulation introduces a trade-off between rendering quality, physical precision, and hardware resources. Achieving 4K photorealism alongside 1ms physics time-steps requires high-end hardware (e.g., RTX 4090), which may limit accessibility for some research groups.
  • Sim-to-Real Challenges: Despite high-fidelity modeling, accurately simulating friction and damping for complex mechanisms like valves remains difficult, highlighting areas for further refinement in physics engines.

Conclusion and Future Work

PR2 establishes a new standard for humanoid simulation by unifying physics and photorealism. Future developments will focus on incorporating built-in Reinforcement Learning (RL) training schemes, automated scene synthesis for increased diversity, and more sophisticated fine-grained grasping capabilities to move beyond symbolic "pick and place" actions.

Summary updated

Abstract

This paper presents the development of a Physics-realistic and Photo-realistic humanoid robot testbed, PR2, to facilitate collaborative research between Embodied Artificial Intelligence (Embodied AI) and robotics. PR2 offers high-quality scene rendering and robot dynamic simulation, enabling (i) the creation of diverse scenes using various digital assets, (ii) the integration of advanced perception or foundation models, and (iii) the implementation of planning and control algorithms for dynamic humanoid robot behaviors based on environmental feedback. The beta version of PR2 has been deployed for the simulation track of a nationwide full-size humanoid robot competition for college students, attracting 137 teams and over 400 participants within four months. This competition covered traditional tasks in bipedal walking, as well as novel challenges in loco-manipulation and language-instruction-based object search, marking a first for public college robotics competitions. A retrospective analysis of the competition suggests that future events should emphasize the integration of locomotion with manipulation and perception. By making the PR2 testbed publicly available at https://github.com/pr2-humanoid/PR2-Platform, we aim to further advance education and training in humanoid robotics. Video demonstration: https://pr2-humanoid.github.io/
BibTeX
@misc{liu2024pr,
  author = {Hangxin Liu and Qi Xie and Zeyu Zhang and Tao Yuan and Xiaokun Leng and Lining Sun and Song-Chun Zhu and Jingwen Zhang and Zhicheng He and Yao Su},
  title = {{PR2: A Physics- and Photo-realistic Testbed for Embodied AI and Humanoid Robots}},
  year = {2024},
  eprint = {2409.01559},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01559},
}
Figure