Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments

Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, Animesh Garg

Introduction

In robot learning, researchers often face a trade-off between simulators optimized for high-fidelity vision (e.g., Habitat) and those optimized for fast physics (e.g., Isaac Gym). This "infrastructure tax" forces developers to spend significant time writing custom code for actuators, sensors, and motion generators. ORBIT addresses this by providing a modular, GPU-accelerated framework built on NVIDIA Isaac Sim, bridging the gap between photo-realistic rendering and complex physics.

System Architecture

ORBIT is designed around two primary abstractions: the World and the Agent. The World defines the physical stage, including robots, sensors, and objects. It incorporates sophisticated actuator models—such as Direct Control motors and Series Elastic Actuators—to simulate real-world constraints like torque saturation and delays, facilitating zero-shot sim-to-real transfer.

ORBIT's modular architecture separating the World (physics/sensors) from the Agent (computation graph).

The Agent is treated as a computation graph. By decoupling perception nodes (e.g., point-cloud generation) from action nodes (e.g., Inverse Kinematics), researchers can swap high-level controllers or motion generators without modifying the underlying physics.

Framework Capabilities and Content

ORBIT provides a "batteries-included" ecosystem, shipping with 16 robotic platforms (including Franka, UR10, and quadrupeds like ANYmal) and over 20 benchmark tasks. These tasks span standard rigid-body manipulation to complex deformable object interactions, such as cloth folding.

Overview of ORBIT's supported sensors, robotic platforms, and diverse object types (rigid, articulated, fluid, and deformable).

Key features include:

  • Deformable Bodies at Scale: Utilizing GPU-based Finite Element Method (FEM) solvers, ORBIT enables large-scale reinforcement learning (RL) for soft-body manipulation.
  • Sensor Management: A flexible system allows sensors (LiDAR, RGB-D) to update at varying frequencies, optimizing performance by avoiding unnecessary rendering at every physics step.
  • Learning Integration: Native support for RL libraries like rl-games and stable-baselines3, and imitation learning via robomimic.

Performance and Validation

The framework leverages PhysX 5.1 and NVIDIA RTX to achieve massive throughput. For rigid body tasks, ORBIT offers a ~10x speedup over CPU-based frameworks, processing 50,000–75,000 frames per second (FPS) across 2048 parallel environments on a single GPU. Physical accuracy was validated through experiments with vibrating soft beams, where FEM-based simulations closely matched real-world silicone damping.

Limitations and Future Work

While physics throughput is high, the rendering engine remains a bottleneck, currently limited to approximately 270 FPS for multiple high-resolution cameras. Future development focuses on further GPU-based rendering acceleration, expanding sensor diversity (e.g., tactile skins), and simplifying the asset pipeline to support direct URDF and OBJ loading alongside the current USD format.

Summary updated

BibTeX
@article{mittal2023orbit,
  author = {Mayank Mittal and Calvin Yu and Qinxi Yu and Jingzhou Liu and Nikita Rudin and David Hoeller and Jia Lin Yuan and Ritvik Singh and Yunrong Guo and Hammad Mazhar and Ajay Mandlekar and Buck Babich and Gavriel State and Marco Hutter and Animesh Garg},
  title = {{Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  doi = {10.1109/lra.2023.3270034},
}
Figure