SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes

Zhili Ng, Haozhe Wang, Zhengshen Zhang, Francis Tay Eng Hock, Marcelo H. Ang Jr

TL;DR

SynTable is an open-source, photorealistic synthetic data generation pipeline built on NVIDIA Isaac Sim that automates the creation of high-quality datasets for amodal instance segmentation. By combining physically-based rendering with automated labeling of complex occlusion relationships, it significantly boosts the real-world performance of computer vision models in cluttered tabletop environments.

What Problem Does This Paper Address?

In the world of computer vision, "amodal instance segmentation" is the challenging task of predicting the full shape of an object, even the parts hidden behind other things. This is a critical skill for robots navigating messy environments—like a kitchen table or a warehouse bin—where they need to understand what they are looking at before they try to grab it.

The field faces three major hurdles:

  1. Data Scarcity: Manually labeling "hidden" parts of objects is slow, expensive, and subjective.
  2. The Sim-to-Real Gap: Most synthetic datasets look "fake" to a neural network. Models trained on low-quality CGI often fail when they see a real-world photo.
  3. Missing Relationships: Existing tools focus on masks but ignore the hierarchy of occlusion (i.e., knowing exactly which object is on top of which).

Methods

SynTable leverages NVIDIA’s Isaac Sim Replicator Composer to create a unified Python-based pipeline. Unlike older simulators that prioritize speed over looks, SynTable uses high-fidelity ray tracing to mimic how light actually behaves in the real world.

The pipeline follows a structured flow:

  1. Scene Preparation: It randomly samples tables and objects, applying domain randomization to textures, materials, and lighting to ensure the model doesn't overfit to specific colors or patterns.
  2. Physics Simulation: Instead of just placing objects, it drops them onto the table using a physics engine, creating natural-looking "clutter."
  3. Multi-View Sampling: It captures the scene from various camera angles and lighting conditions.
  4. Automated Annotation: This is the "secret sauce." The tool automatically generates modal (visible) masks, amodal (full) masks, depth maps, and—crucially—an Occlusion Order Adjacency Matrix (OOAM).
High-level overview of the SynTable synthetic data generation pipeline.

The authors also introduced a new metric, Occlusion Order Accuracy ($ACC_{OO}$), to quantify how well a model understands the "stacking" order of objects in a scene.

Datasets

The authors released SynTable-Sim, a massive dataset generated with their tool:

  • Scale: 50,000 RGB-D images across 1,000 unique scenes.
  • Diversity: 1,075 novel object models sourced from the Google Scanned Objects and BOP datasets.
  • Complexity: It features a much higher "occlusion rate" (17.56%) compared to previous synthetic datasets like UOAIS-Sim (11.16%), making it a tougher and more realistic training ground.

For evaluation, they tested their models on the OSD-Amodal dataset, which consists of real-world images of cluttered tables, to measure how well the synthetic training transferred to reality.

Key Insights and Ideas

  • Photorealism is Non-Negotiable: The jump in performance suggests that ray-traced lighting and physically accurate materials are the primary drivers for successful sim-to-real transfer in amodal tasks.
  • Physics Creates Better Labels: By dropping objects rather than placing them, the pipeline generates complex, multi-layered occlusions that are nearly impossible for humans to label consistently but are vital for training robust models.
  • Occlusion as a Graph: By treating occlusion as a directed graph (OODAG), the authors move beyond simple pixel-pushing toward a more structural understanding of 3D scenes. This is a "lightbulb moment" for robotics, where knowing the order of objects is the difference between a successful pick and a crashed arm.
The process of capturing rich ground truth annotations, including the occlusion order graph.

Key Results

The results were striking. When training the state-of-the-art UOAIS-Net model:

  • Real-World Accuracy: On the real-world OSD-Amodal test set, the model trained on SynTable-Sim achieved an Overlap F-measure of 80.9, nearly doubling the 42.4 achieved by the previous best synthetic dataset (UOAIS-Sim).
  • Occlusion Reasoning: The new $ACC_{OO}$ metric showed a massive leap from 12.7 to 82.9, proving the model finally "understood" which objects were on top.
  • Efficiency: Even when using only half the SynTable-Sim data, the model still significantly outperformed the full baseline dataset.

Limitations and Future Work

While SynTable is a powerful tool, the authors note it is currently optimized for tabletop scenes. While the framework is generalizable, expanding it to larger-scale environments (like entire rooms or outdoor scenes) would require new sets of assets and environmental parameters.

Additionally, the quality of the output is still dependent on the quality of the 3D CAD models provided. Future work could involve integrating more dynamic scenes or complex object-object interactions beyond simple stacking.

Summary updated

Abstract

In this work, we present SynTable, a unified and flexible Python-based dataset generator built using NVIDIA's Isaac Sim Replicator Composer for generating high-quality synthetic datasets for unseen object amodal instance segmentation of cluttered tabletop scenes. Our dataset generation tool can render complex 3D scenes containing object meshes, materials, textures, lighting, and backgrounds. Metadata, such as modal and amodal instance segmentation masks, object amodal RGBA instances, occlusion masks, depth maps, bounding boxes, and material properties can be automatically generated to annotate the scene according to the users' requirements. Our tool eliminates the need for manual labeling in the dataset generation process while ensuring the quality and accuracy of the dataset. In this work, we discuss our design goals, framework architecture, and the performance of our tool. We demonstrate the use of a sample dataset generated using SynTable for training a state-of-the-art model, UOAIS-Net. Our state-of-the-art results show significantly improved performance in Sim-to-Real transfer when evaluated on the OSD-Amodal dataset. We offer this tool as an open-source, easy-to-use, photorealistic dataset generator for advancing research in deep learning and synthetic data generation. The links to our source code, demonstration video, and sample dataset can be found in the supplementary materials.
BibTeX
@inproceedings{ng2025syntable,
  author = {Zhili Ng and Haozhe Wang and Zhengshen Zhang and Francis Tay Eng Hock and Marcelo H. Ang Jr},
  title = {{SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes}},
  booktitle = {SynData4CV @ CVPR},
  year = {2025},
  url = {https://openreview.net/forum?id=L7xL80JclK},
}
Figure