Introduction
The "Sim2Real" transfer problem remains a significant challenge in robotics. Reinforcement learning (RL) agents trained in simulation often fail in physical environments due to the "reality gap"—the discrepancy between simulated and real-world sensory inputs. This gap is particularly pronounced in computer vision, where unpredictable lighting, textures, and optical phenomena complicate task execution.
Domain Randomization (DR) has emerged as a robust solution, involving the training of agents across a vast array of randomized simulated environments. However, traditional game engines are optimized for human-centric experiences, prioritizing low latency and high resolution. In contrast, machine learning research requires high-throughput batch rendering of low-resolution images across distributed, headless cloud environments. To address this, OpenAI developed the Remote Rendering Backend (ORRB).
System Design and Implementation
ORRB is a high-throughput, modular rendering system built on Unity3D (for visual fidelity) and MuJoCo (for physics simulation). It operates as a standalone rendering service, facilitating the massive-scale generation of synthetic data.
Key Technical Features
- Renderer as a Service: Utilizing a gRPC interface, ORRB is language-agnostic, allowing seamless integration with Python-based ML frameworks.
- Single Scene Multiple States (SSMS): To maximize efficiency, ORRB shares heavy assets like geometry and textures across multiple instances. It only updates lightweight randomized parameters for each state within a batch, significantly reducing overhead.
- Modular Component Manager: The system features a "plug-and-play" architecture for randomizers. Users can attach components to randomize materials, camera jitter, lighting intensity, and post-processing effects such as grain and bloom.
- Optimized Capture Pipeline: To prevent CPU/GPU stalls, ORRB employs a round-robin pool of render textures and bulk DMA transfers to move completed batches back to CPU memory.
Experimental Setup and Evaluation
The system was evaluated using the environment from OpenAI’s Dactyl project, which involves a Shadow Dexterous Hand manipulating a cube. ORRB processes MuJoCo XML scene descriptions and generates multi-channel data, including RGB images, depth maps, segmentation masks, and screen-space normal vectors.
Performance and Scaling
Benchmarks conducted on Google Cloud Platform using NVIDIA V100 GPUs highlight ORRB's efficiency:
- Throughput: A single V100 GPU achieved a throughput of 1,342 frames per second (FPS).
- Scaling: Performance scales effectively with multiple GPUs, reaching 3,514 FPS on a 4-GPU configuration.
- Saturation: Optimal performance is reached when 16 to 24 parallel render servers saturate a single V100 GPU.
A key insight from these results is that the CPU often becomes the bottleneck. Since ML tasks typically require low-resolution images (e.g., 224x224), the GPU completes rendering tasks almost instantly, leaving the CPU to handle the overhead of scene setup and rendering call management.
Conclusion and Future Work
ORRB demonstrates that prioritizing throughput over latency is essential for large-scale robotic simulation. While the system successfully bridges the reality gap for complex tasks, some limitations remain. Unity’s lack of native headless rendering support on Linux required workarounds using Xorg instances. Additionally, CPU scaling limits were observed beyond three GPUs per node due to scene setup overhead. Future work may address these bottlenecks and the minor non-determinism inherent in GPU-based rendering.
ORRB is open-source and available at github.com/openai/orrb.
Summary updated