scene_synthesizer: A Python Library for Procedural Scene Generation in Robot Manipulation

Clemens Eppner, Adithyavairavan Murali, Caelan Reed Garrett, Rowland O'Flaherty, Tucker Hermans, Wei Yang, Dieter Fox, Rowland O’FlahertyJ. Open Source Softw 2025 · DOI · DBLP

Abstract

Training robust deep learning models for robotics requires massive amounts of diverse, physically grounded data. scene_synthesizer is an open-source Python library designed to address the bottleneck of environment creation by procedurally generating complex 3D scenes. By decoupling scene logic from specific physics engines, it allows researchers to create diverse, articulated environments—such as kitchens—and export them seamlessly to simulators including MuJoCo, PyBullet, and Isaac Sim.

Introduction

The development of generalizable robotic agents is often hindered by the limitations of existing simulation tools. Current challenges include the lack of diversity in static datasets, the absence of physical metadata (e.g., mass, friction) in many generative 3D models, and "simulator lock-in," where generation pipelines are tied to a single physics engine. scene_synthesizer addresses these gaps by providing a simulator-agnostic middleware that facilitates the creation of high-quality, physically realistic virtual environments at scale.

Methodology

scene_synthesizer leverages the trimesh library for geometric representation and provides a high-level Python API for scripting scene generation.

Key Technical Features

  • Procedural Asset Library: The library includes over 30 parameterizable, kitchen-themed objects that can be instantiated dynamically.
  • Smart Placement Logic: Instead of manual coordinate entry, the library uses automated labeling of support surfaces and containment volumes. This allows for "object-agnostic" placement, such as automatically positioning items inside drawers or on tabletops regardless of specific geometry.
  • Physical Grounding and Articulation: Users can define comprehensive physical properties, including collision geometry, mass, and friction. Crucially, the library supports multi-part objects with functional joints (e.g., cabinets, appliances), complete with stiffness and damping parameters.
Comparison between semantic segmentation (left) and a physically grounded render (right) of a procedurally generated kitchen.

Results and Applications

The utility of scene_synthesizer is demonstrated through its adoption in various research projects, including neural motion planners, collision detection systems (CabiNet), and vision-language model (VLM) fine-tuning. A core strength of the library is its "write once, simulate anywhere" capability.

A synthetic kitchen scene rendered across multiple platforms and formats, demonstrating the library's simulator-agnostic export capabilities.

As shown in the figure above, a single synthetic scene can be successfully exported to formats like USD and URDF, allowing for consistent simulation and rendering across diverse platforms such as PyBullet, MuJoCo, and Isaac Sim. This enables "sim-to-sim" validation, ensuring that learned robotic behaviors are robust and not merely artifacts of a specific engine's physics implementation.

Discussion and Limitations

By focusing on the kitchen domain, scene_synthesizer tackles the "grand challenge" of domestic robotics, characterized by high clutter and articulated objects. The library's procedural logic significantly reduces the manual effort required to generate complex environments.

However, several areas for future development remain. The current asset library is heavily biased toward kitchen environments; expanding this to include warehouses or medical settings would broaden its applicability. Additionally, while the library is lightweight, it currently requires Python proficiency, and the diversity of its procedural assets must continue to grow to match the variety found in massive static datasets like PartNet.

Summary updated

BibTeX
@article{eppner2025scene,
  author = {Clemens Eppner and Adithyavairavan Murali and Caelan Reed Garrett and Rowland O'Flaherty and Tucker Hermans and Wei Yang and Dieter Fox and Adithyavairavan Murali and Rowland O’Flaherty and Clemens Eppner},
  title = {{scene\_synthesizer: A Python Library for Procedural Scene Generation in Robot Manipulation}},
  journal = {J. Open Source Softw.},
  year = {2025},
  doi = {10.21105/joss.07561},
}
Figure