Introduction
In robot learning, researchers often face a trade-off between simulators optimized for high-fidelity vision (e.g., Habitat) and those optimized for fast physics (e.g., Isaac Gym). This "infrastructure tax" forces developers to spend significant time writing custom code for actuators, sensors, and motion generators. ORBIT addresses this by providing a modular, GPU-accelerated framework built on NVIDIA Isaac Sim, bridging the gap between photo-realistic rendering and complex physics.
System Architecture
ORBIT is designed around two primary abstractions: the World and the Agent. The World defines the physical stage, including robots, sensors, and objects. It incorporates sophisticated actuator models—such as Direct Control motors and Series Elastic Actuators—to simulate real-world constraints like torque saturation and delays, facilitating zero-shot sim-to-real transfer.
The Agent is treated as a computation graph. By decoupling perception nodes (e.g., point-cloud generation) from action nodes (e.g., Inverse Kinematics), researchers can swap high-level controllers or motion generators without modifying the underlying physics.
Framework Capabilities and Content
ORBIT provides a "batteries-included" ecosystem, shipping with 16 robotic platforms (including Franka, UR10, and quadrupeds like ANYmal) and over 20 benchmark tasks. These tasks span standard rigid-body manipulation to complex deformable object interactions, such as cloth folding.
Key features include:
- Deformable Bodies at Scale: Utilizing GPU-based Finite Element Method (FEM) solvers, ORBIT enables large-scale reinforcement learning (RL) for soft-body manipulation.
- Sensor Management: A flexible system allows sensors (LiDAR, RGB-D) to update at varying frequencies, optimizing performance by avoiding unnecessary rendering at every physics step.
- Learning Integration: Native support for RL libraries like
rl-gamesandstable-baselines3, and imitation learning viarobomimic.
Performance and Validation
The framework leverages PhysX 5.1 and NVIDIA RTX to achieve massive throughput. For rigid body tasks, ORBIT offers a ~10x speedup over CPU-based frameworks, processing 50,000–75,000 frames per second (FPS) across 2048 parallel environments on a single GPU. Physical accuracy was validated through experiments with vibrating soft beams, where FEM-based simulations closely matched real-world silicone damping.
Limitations and Future Work
While physics throughput is high, the rendering engine remains a bottleneck, currently limited to approximately 270 FPS for multiple high-resolution cameras. Future development focuses on further GPU-based rendering acceleration, expanding sensor diversity (e.g., tactile skins), and simplifying the asset pipeline to support direct URDF and OBJ loading alongside the current USD format.
Summary updated