Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics

Max Schwarz, Sven Behnke

Training deep learning models for robotics is a notorious "chicken and egg" problem. To get a robot to perceive its environment, you need massive amounts of labeled data. But to get that data, you often need to manually build scenes, capture images, and painstakingly annotate them—a process that is slow, expensive, and doesn't scale when the robot encounters a new object or environment.

In their paper "Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics," Max Schwarz and Sven Behnke from the University of Bonn present a compelling solution: an open-source library that generates physically realistic, cluttered 3D scenes on the fly to train perception models without a single real-world training image.

TL;DR

Stillleben is an online scene synthesis pipeline that uses physics simulation and advanced rendering to create training data for robotic perception. By simulating how objects fall and interact, and then imbuing the renders with realistic camera noise, it allows models to achieve ~86% of the performance of real-world datasets on tasks like semantic segmentation.

What Problem Does This Paper Address?

Roboticists often work in "specialized domains"—think a specific warehouse shelf or a home kitchen. Existing large-scale datasets (like COCO or ImageNet) don't cover these specific objects, and 2D synthesis (simply "pasting" object images onto backgrounds) fails to capture the nuances of 3D space: realistic lighting, shadows, and physical interactions like stacking and occlusion.

The authors aim to eliminate the need for manual data collection by creating a "digital twin" of the training process. The challenge is the reality gap: if the synthetic data isn't "realistic" enough in terms of physics and sensor noise, the model will fail when deployed on a real robot.

Methods

The Stillleben (German for "still life") pipeline consists of three main stages:

  1. Arrangement Engine (Physics): Instead of placing objects at random coordinates, the system uses the NVIDIA PhysX engine. It "drops" object meshes onto a supporting plane. This ensures that objects rest on each other naturally, creating the kind of dense clutter and complex occlusions found in real-world robotic bins.
  2. Rendering (Visuals): Using the Magnum OpenGL engine, the system applies:
  3. Physically-based Rendering (PBR): Realistic material properties like "metalness" and "roughness."
    • Image-based Lighting (IBL): Using real-world environment maps to create complex reflections.
  4. Sticker Projection: A clever augmentation that randomly projects "product labels" or barcodes onto meshes to simulate the visual noise of real-world packaging.
  5. Camera Model (Sensor Simulation): This is the final "polish." The system simulates commodity camera artifacts like chromatic aberration, blur, exposure limits, and sensor noise. Crucially, this happens on the GPU using PyTorch/CUDA, keeping the pipeline fast.

Datasets

The authors evaluated their system using:

  • YCB-Video Dataset: A standard benchmark for robotic object pose and segmentation.
  • Falling Things (FAT): A high-quality offline synthetic dataset used for comparison.
  • Sketchfab: Used to source a variety of "driller" meshes to test how well the system generalizes to object categories rather than just specific instances.
  • ObjectNet3D: Used for sourcing background images to place behind the synthetic clutter.

Key Insights and Ideas

  • The "Camera Model" is Essential: In their ablation study, removing the camera noise simulation caused the Mean IoU (Intersection over Union) to plummet from 0.656 to 0.128. This suggests that "perfect" renders are actually bad for training; the model needs to learn to see through the "grime" of real-world sensors.
  • Physics-Based Clutter Matters: By specializing in dense, cluttered arrangements, Stillleben outperformed the FAT dataset, which features more spread-out objects. Realistic occlusion is a feature, not a bug, for training robust models.
  • Online Synthesis vs. Static Datasets: Because Stillleben is fast (30 fps for generation), it can run during training. The neural network never sees the same image twice, which acts as a powerful form of regularization and prevents overfitting to synthetic artifacts.

Key Results

  • Near-Real Performance: A model trained only on Stillleben data achieved a 0.656 Mean IoU on the YCB-Video test set. For context, a model trained on the actual real-world training set achieved 0.760. Reaching ~86% of "ground truth" performance with zero manual labels is a significant win.
  • Efficiency: The pipeline generates 640x480 annotated images at 30 fps. When integrated into a training loop (RefineNet), the entire system runs at 10 fps.
  • Real-World Validation: The authors used the pipeline to train a robot for a "regrasping" task involving watering cans and spray bottles. The model achieved a 100% segmentation success rate in real-world trials, leading to a 65% overall task success rate in a complex manipulation pipeline.

Limitations and Future Work

  • Mesh Dependency: The system still requires high-quality 3D meshes. While 3D scanning is getting easier, it remains a bottleneck for novel objects.
  • Functional Bias: The physics engine produces "random" piles. In reality, humans place objects with intent (e.g., a cup is usually upright). The authors suggest that future work could involve more "functional" arrangement options to capture this human bias.
  • The Remaining Gap: There is still a ~10% performance gap between synthetic-only and real-world training. Bridging this final gap likely requires even more sophisticated material modeling or domain adaptation techniques.

Check out the project: The authors have released Stillleben as an open-source library with PyTorch bindings. You can find it at github.com/AIS-Bonn/stillleben.

Summary updated

Abstract

Training data is the key ingredient for deep learning approaches, but difficult to obtain for the specialized domains often encountered in robotics. We describe a synthesis pipeline capable of producing training data for cluttered scene perception tasks such as semantic segmentation, object detection, and correspondence or pose estimation. Our approach arranges object meshes in physically realistic, dense scenes using physics simulation. The arranged scenes are rendered using high-quality rasterization with randomized appearance and material parameters. Noise and other transformations introduced by the camera sensors are simulated. Our pipeline can be run online during training of a deep neural network, yielding applications in life-long learning and in iterative render-and-compare approaches. We demonstrate the usability by learning semantic segmentation on the challenging YCB-Video dataset without actually using any training frames, where our method achieves performance comparable to a conventionally trained model. Additionally, we show successful application in a real-world regrasping system.
BibTeX
@inproceedings{schwarz2020stillleben,
  author = {Max Schwarz and Sven Behnke},
  title = {{Stillleben: Realistic Scene Synthesis for Deep Learning in Robotics}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=JQKiQTleRm6},
}

Loading related content...

Figure