Silver: Novel Rendering Engine for Data Hungry Computer Vision Models

Abdulrahman Kerim, Leandro Soriano Marcolino, Richard JiangLancaster 2021

Abstract

Large-scale synthetic data is needed to support the deep learning big-bang that started in the recent decade and influenced almost all scientific fields. Most of the synthetic data generation solutions are task-specific or unscalable while the others are expensive, based on commercial games, or unreliable. In this work, a new rendering engine called Silver is presented in detail. Photo-realism, diversity, scalability, and full 3D virtual world generation at run-time are the key aspects of this work. The photo-realism was approached by utilizing the state-of-the-art High Definition Render Pipeline (HDRP) of the Unity game engine. In parallel, the Procedural Content Generation (PCG) concept was employed to create a full 3D virtual world at run-time, while the scalability of the system was attained by taking advantage of the modular approach followed as we built the system from scratch. Silver can be used to provide clean, unbiased, and large-scale training and testing data for various computer vision tasks.

Introduction

The rapid advancement of deep learning has created an insatiable demand for massive, accurately labeled datasets. However, gathering real-world data presents significant logistical challenges: manual labeling is expensive and error-prone, and capturing rare "edge cases"—such as specific accidents or extreme weather—is often dangerous or impossible. Furthermore, tightening privacy regulations make the collection and sharing of real-world imagery increasingly difficult.

While synthetic data offers a promising alternative, existing solutions often suffer from limited scalability, lack of visual fidelity (the "domain gap"), or reliance on closed commercial game engines like Grand Theft Auto V that are difficult to modify for research purposes. Silver addresses these gaps by providing a modular, photorealistic, and procedurally-driven engine built specifically for computer vision research.

System Architecture and Methods

Silver is built upon Unity’s High Definition Render Pipeline (HDRP), enabling physically-based lighting and materials that ensure interactions between light and surfaces follow the laws of physics. This is crucial for models to generalize to real-world optical conditions.

The engine's architecture is divided into several modular controllers, as illustrated in the hierarchical structure below:

The hierarchical architecture of the Silver Scene Generator, detailing static/dynamic elements, camera control, weather systems, and ground-truth extraction.

Procedural Content Generation (PCG)

Rather than relying on static, pre-built maps, Silver generates 3D worlds at run-time. It procedurally determines environmental parameters such as street lengths, intersection counts, and building placements. This "online" generation ensures that every sequence is unique, preventing models from over-fitting to specific background geometries.

Automated Ground-Truth Extraction

A core feature of Silver is its custom Ground-truth Extractor. Using specialized shaders, the engine produces pixel-perfect annotations simultaneously with RGB frames. Supported outputs include semantic and instance segmentation, depth maps, surface normals, and 14-joint human body pose coordinates.

Key Results and Qualitative Analysis

Silver demonstrates high performance and visual fidelity. On a GeForce GTX 1080 Ti, the system maintains approximately 30 FPS for moderately crowded scenes. Quantitative analysis shows that the engine handles high-density "crowd" scenarios efficiently, with rendering overhead (SetPass Calls) remaining stable even as the number of polygons increases.

Visually, Silver produces images with higher fidelity and more complex environmental effects (like night-time lighting and realistic fog) than established generators like PHAV or Synthia.

A sample render from the Silver engine demonstrating high-fidelity urban environments with realistic lighting, materials, and dynamic objects.

Conclusion and Future Work

Silver provides a scalable, future-proof tool for generating unbiased, large-scale training data. By treating data generation as an adaptive, run-time process, it opens new avenues for integration with online learning algorithms.

Future developments will focus on refining the probability distributions of randomized elements (e.g., matching car color frequencies to real-world statistics) and increasing the behavioral complexity of agents to move beyond simple population toward truly interactive environments. Additionally, further quantitative studies are planned to measure the exact improvement in real-world benchmark performance when models are trained using Silver-generated data.

Summary updated

Abstract

Large-scale synthetic data is needed to support the deep learning big-bang that started in the recent decade and influenced almost all scientific fields. Most of the synthetic data generation solutions are task-specific or unscalable while the others are expensive, based on commercial games, or unreliable. In this work, a new rendering engine called Silver is presented in detail. Photo-realism, diversity, scalability, and full 3D virtual world generation at run-time are the key aspects of this work. The photo-realism was approached by utilizing the state-of-the-art High Definition Render Pipeline (HDRP) of the Unity game engine. In parallel, the Procedural Content Generation (PCG) concept was employed to create a full 3D virtual world at run-time, while the scalability of the system was attained by taking advantage of the modular approach followed as we built the system from scratch. Silver can be used to provide clean, unbiased, and large-scale training and testing data for various computer vision tasks.
BibTeX
@inproceedings{kerim2021silver,
  author = {Abdulrahman Kerim and Leandro Soriano Marcolino and Richard Jiang},
  title = {{Silver: Novel Rendering Engine for Data Hungry Computer Vision Models}},
  booktitle = {Lancaster},
  year = {2021},
}
Figure