SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model

Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, Yunfeng Ai

TL;DR

This paper introduces SimWorld, a novel pipeline for generating realistic autonomous driving scenes by combining simulation engines with world models. It addresses the critical challenge of data scarcity for rare driving scenarios by creating a unified benchmark that bridges the gap between synthetic and real-world data, significantly improving downstream perception model performance.

What Problem Does This Paper Address?

The rapid advancement of autonomous driving technology has created an insatiable demand for high-quality, diverse datasets. However, collecting comprehensive datasets, especially those covering rare "corner cases" (e.g., extreme weather, sudden road changes), is prohibitively expensive and time-consuming. Existing methods for generating synthetic data often suffer from a significant visual gap with real-world data, hindering the generalization capabilities of autonomous driving systems. While world models offer a promising avenue for controllable data generation, they are often limited by the scarcity of corner cases in their training data, preventing them from generating truly novel scenarios. Furthermore, general simulators can introduce distribution differences, and adapting existing simulators for specific needs can be complex and inflexible.

Methods

SimWorld proposes a novel simulator-conditioned scene generation pipeline that leverages the strengths of both simulation engines and world models. The core idea is to use a simulator that is highly consistent with real-world scenes to generate "conditions" (like ground truth labels, bounding boxes, and semantic information) that then guide a world model (specifically, a diffusion model) to generate realistic scenes.

Key components include:

  • Simulator-Conditioned Scene Generation: Instead of relying solely on real-world data or general simulators, SimWorld uses a specialized simulator (PMWorld) that is meticulously engineered to match real-world mining environments. This simulator provides rich, accurate labels and conditions.
  • World Model Integration: A diffusion model, enhanced with ControlNet, is employed for scene generation. ControlNet allows for the integration of various conditional inputs (like bounding boxes and semantic masks) to guide the diffusion process.
  • Dynamic Foreground Weight Loss: A novel loss function is introduced to dynamically adjust the importance of foreground elements during training. This helps the model focus on key objects and refine image details, improving overall generation quality.
  • Prompt Engineering for Conditions: Detection labels are converted into natural language descriptions to provide richer, more interpretable conditions for the diffusion model, enhancing its ability to generate diverse and accurate scenes.
  • Simulation-to-Reality Conversion: The inference process uses conditions sampled from the simulator to generate scenes, effectively bridging the simulation-to-reality gap.

Datasets

  • Training Data: The primary training data is derived from the AutoMine dataset, augmented with techniques like horizontal flipping and hue adjustments to create a dataset of 32,000 samples. Each sample includes scene images, segmentation masks, detection boxes, and prompt-generated descriptions.
  • Inference Data: The PMScenes dataset (11,000 samples) is used for inference, preprocessed to match the real mining data format.
  • Urban Data Exploration: The Cityscapes dataset is used to demonstrate the approach's applicability beyond mining scenes, although it's noted that the dataset's size (3.4k training samples) is relatively small for large-scale generation.
  • Benchmark: The paper introduces SimWorld as a unified benchmark for simulator-conditioned scene generation, providing both real and virtual data for evaluation.

Key Insights and Ideas

  • Simulator Consistency is Key: The paper strongly emphasizes that the fidelity and consistency of the simulator with real-world environments are crucial for generating high-quality, transferable synthetic data.
  • Bridging the Sim-to-Real Gap: By using a highly realistic simulator to generate conditions for a powerful world model, SimWorld effectively mitigates the visual domain gap that plagues many existing synthetic data generation methods.
  • Data Augmentation via Simulation: SimWorld demonstrates that simulation can be a powerful tool for generating diverse and challenging scenarios that are difficult or impossible to capture in the real world, thus addressing data scarcity for corner cases.
  • Conditional Generation for Control: The use of ControlNet and prompt engineering allows for fine-grained control over the generated scenes, enabling the creation of specific scenarios tailored to downstream tasks.

Key Results

  • Improved Perception Performance: Experiments show that perception models trained with SimWorld-generated data significantly outperform those trained with random initialization, public datasets (like KITTI), and even other synthetic datasets (like PMScenes). For example, in object detection, the SimWorld-pretrained strategy achieved the best performance across various algorithms.
  • High-Quality and Diverse Generation: Quantitative evaluation using FID and Dpix metrics indicates that SimWorld generates images with a data distribution closer to real-world scenes and exhibits greater pixel-level diversity.
  • Scalability: The proposed pipeline is designed for large-scale data generation, addressing a key limitation of previous methods that prioritized quality over quantity.
  • Urban Scene Potential: While primarily focused on mining, the approach shows promising results on urban scenes (Cityscapes), suggesting its broader applicability.

Limitations and Future Work

  • Computational Cost: Training larger models like SimWorld XL requires significant computational resources (e.g., multiple high-end GPUs over extended periods).
  • Dataset Size for Urban Scenes: The paper notes that the Cityscapes dataset is relatively small for large-scale generation, implying that more extensive urban datasets would be beneficial for further exploration.
  • Future Work: The authors suggest exploring the generative performance of the world model in urban autonomous driving scenarios more extensively. The availability of their data and code is intended to facilitate further research in this area.

Summary updated

Abstract

With the rapid advancement of autonomous driving technology, a lack of data has become a major obstacle to enhancing perception model accuracy. Researchers are now exploring controllable data generation using world models to diversify datasets. However, previous work has been limited to studying image generation quality on specific public datasets. There is still relatively little research on how to build data generation engines for real-world application scenes to achieve large-scale data generation for challenging scenes. In this paper, a simulator-conditioned scene generation engine based on world model is proposed. By constructing a simulation system consistent with real-world scenes, simulation data and labels, which serve as the conditions for data generation in the world model, for any scenes can be collected. It is a novel data generation pipeline by combining the powerful scene simulation capabilities of the simulation engine with the robust data generation capabilities of the world model. In addition, a benchmark with proportionally constructed virtual and real data, is provided for exploring the capabilities of world models in real-world scenes. Quantitative results show that these generated images significantly improve downstream perception models performance. Finally, we explored the generative performance of the world model in urban autonomous driving scenarios. All the data and code will be available at https://github.com/Li-Zn-H/SimWorld.
BibTeX
@inproceedings{li2025simworld,
  author = {Xinqing Li and Ruiqi Song and Qingyu Xie and Ye Wu and Nanxin Zeng and Yunfeng Ai},
  title = {{SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=uajoq20q6m},
}
Figure