Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

Max Schwarz, Sven Behnke

Abstract

Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduced by the camera sensors are simulated. Our pipeline can be run online during training of a deep neural network, yielding applications in life-long learning and in iterative render-and-compare approaches. We demonstrate the usability by learning semantic segmentation on the challenging YCB-Video dataset without actually using any training frames, where our method achieves performance comparable to a conventionally trained model. Additionally, we show successful application in a real-world regrasping system.

Introduction

Robotic perception in specialized domains—such as warehouse logistics or domestic kitchens—is often hampered by a lack of labeled training data. While large-scale datasets like COCO or ImageNet exist, they rarely cover the specific objects or dense, cluttered environments encountered in robotics. Manually capturing and annotating such data is slow and expensive. To address this, the Stillleben (German for "still life") library provides an open-source pipeline for generating physically realistic, cluttered 3D scenes on the fly, bridging the "reality gap" between synthetic training and real-world deployment.

Methodology: The Stillleben Pipeline

The Stillleben synthesis pipeline: from object meshes to a training frame.

The Stillleben pipeline consists of three primary stages designed to maximize realism and training efficiency:

  1. Arrangement Engine (CPU): Utilizing the NVIDIA PhysX engine, the system simulates physics by "dropping" object meshes onto a supporting plane. This ensures that objects interact naturally, creating realistic piles with complex occlusions and stacking behaviors characteristic of robotic bins.
  2. Rendering (GPU): The Magnum OpenGL engine is used for high-quality rendering. It incorporates Physically-based Rendering (PBR) for material properties like metalness and roughness, and Image-based Lighting (IBL) using real-world environment maps. A unique "sticker projection" augmentation randomly projects labels or barcodes onto meshes to simulate real-world packaging noise.
  3. Camera Model (GPU): To further bridge the reality gap, the system simulates commodity camera artifacts, including chromatic aberration, blur, exposure limits, and sensor noise. This stage is implemented on the GPU using PyTorch/CUDA to maintain high throughput.

Crucially, Stillleben is designed for online synthesis. Generating 640x480 annotated images at 30 fps allows the pipeline to run during the training of a neural network. This ensures the model never sees the same image twice, acting as a powerful regularizer.

Experimental Results and Discussion

A sample synthetic scene generated by the pipeline, featuring realistic occlusion and lighting.

The effectiveness of the pipeline was evaluated on the YCB-Video benchmark. A model trained exclusively on Stillleben synthetic data achieved a 0.656 Mean IoU (Intersection over Union) on the test set, reaching approximately 86% of the performance of a model trained on real-world data (0.760 mIoU).

Ablation studies highlighted the critical importance of the camera model; removing sensor noise simulation caused the Mean IoU to drop from 0.656 to 0.128. This underscores that "perfect" synthetic renders are often detrimental to training robust real-world models. Furthermore, the physics-based arrangement outperformed spread-out synthetic datasets like Falling Things (FAT), proving that realistic occlusion is a vital feature for training.

Beyond benchmarks, the pipeline was validated in a real-world robotic regrasping task. The trained model achieved a 100% segmentation success rate in trials involving watering cans and spray bottles, contributing to a 65% overall task success rate in complex manipulation scenarios.

Conclusion

Stillleben demonstrates that high-quality, physics-informed scene synthesis can significantly reduce the dependency on manual data collection in robotics. While a performance gap remains compared to real-world training, the system's ability to generalize to novel objects and its efficiency in online training make it a powerful tool for robotic perception. Future work may focus on more functional object arrangements and even more sophisticated material modeling to close the remaining reality gap.

Summary updated

Abstract

Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduced by the camera sensors are simulated. Our pipeline can be run online during training of a deep neural network, yielding applications in life-long learning and in iterative render-and-compare approaches. We demonstrate the usability by learning semantic segmentation on the challenging YCB-Video dataset without actually using any training frames, where our method achieves performance comparable to a conventionally trained model. Additionally, we show successful application in a real-world regrasping system.
BibTeX
@inproceedings{schwarz2020stillleben,
  author = {Max Schwarz and Sven Behnke},
  title = {{Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=JQKiQTleRm6},
}
Figure