Depth-SIMS: Semi-Parametric Image and Depth Synthesis

Valentina Musat, Daniele De Martini, Matthew Gadd, Paul Newman

Abstract

In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increase in mIoU over SOTA by 3.7 percentage points and a highly competitive FID. Furthermore, we analyse the quality of the generated data as training data for semantic segmentation and depth completion, and show that our approach is more suited for this purpose than other methods.

Introduction

Training deep learning models for autonomous driving requires massive amounts of pixel-perfect ground truth (GT) data. However, manual annotation is notoriously expensive and slow. While simulators like CARLA or pure generative models (GANs) offer alternatives, they often suffer from a "reality gap" or structural inconsistencies where the generated image does not perfectly match the underlying semantic labels.

Previous semi-parametric methods, such as SIMS, attempted to bridge this gap by stitching real image "blobs" onto a canvas. Yet, these methods faced two primary hurdles: maintaining perfect alignment between the synthesized image and the segmentation mask, and the lack of multi-modal output, specifically depth information. Depth-SIMS addresses these challenges by generating high-quality RGB-D data through a novel four-stage pipeline.

Methodology

Depth-SIMS utilizes a semi-parametric framework that combines real-world image fragments with neural synthesis:

  1. Object Blob Database: The system decomposes the Cityscapes dataset into a library of object "blobs" (e.g., cars, pedestrians). To ensure efficient retrieval, the authors use Hu moments—mathematical shape descriptors—to quickly find real-world objects that fit a target layout.
  2. Canvasing: Using a guiding layout, blobs are pasted onto a blank canvas. A smart hole-filling strategy is applied to the segmentation map to ensure no empty pixels remain, maintaining structural integrity.
  3. Sparse Depth Composition: Rather than focusing solely on color, the system samples depth values from the original source data to create a sparse depth map perfectly aligned with the new RGB canvas.
  4. In-painting and Synthesis: A GAN-based in-painter, utilizing the SPADE architecture and an OASIS segmenting discriminator, performs dual-task synthesis. It blends the edges of pasted blobs for visual naturalism while simultaneously "densifying" the sparse depth map into a complete, pixel-wise depth image.

Experimental Results

Structural Alignment and Realism

A key finding of this research is that for training downstream models, structural alignment is more critical than pure perceptual realism. Depth-SIMS achieved a 3.7 percentage point increase in mIoU over previous state-of-the-art methods, indicating a significantly tighter fit between images and labels. Despite this focus on alignment, the model maintained a competitive FID score of 46.3, outperforming many fully parametric GANs.

Downstream Task Performance

The utility of Depth-SIMS data was tested on several benchmarks:

  • Generalization: Models trained on Depth-SIMS data performed better on the A2D2 (Audi Autonomous Driving Dataset) than those trained on SIMS or OASIS data, demonstrating superior out-of-domain generalization.
  • Depth Completion: On the KITTI dataset, the synthesized depth maps achieved a Root Mean Squared Error (RMSE) of 3.90m, a substantial improvement over the 5.99m RMSE produced by raw guiding depth.

Discussion and Limitations

By forcing the network to predict RGB and Depth simultaneously, the model learns a more robust representation of scene geometry, which inherently improves image quality. Furthermore, replacing standard pair-wise IoU matching with Hu moments significantly accelerates the retrieval process without sacrificing accuracy.

However, some limitations remain. The current system relies on a heuristic ordering (e.g., pasting background elements before foreground objects) to handle occlusions. Additionally, the distinction between "static" classes (like vegetation) and "dynamic" classes (like cars) is manual. Future work may involve learning object ordering and expanding the blob database to include more diverse environments to reduce source dependency.

Conclusion

Depth-SIMS demonstrates that a semi-parametric approach, prioritizing structural alignment and multi-modal synthesis, provides a more effective path for generating synthetic training data than traditional generative methods. By providing high-quality, aligned RGB-D images, it offers a scalable solution for the data-hungry requirements of autonomous driving stacks.

Summary updated

Abstract

In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increase in mIoU over SOTA by 3.7 percentage points and a highly competitive FID. Furthermore, we analyse the quality of the generated data as training data for semantic segmentation and depth completion, and show that our approach is more suited for this purpose than other methods.
BibTeX
@inproceedings{musat2022depth,
  author = {Valentina Musat and Daniele De Martini and Matthew Gadd and Paul Newman},
  title = {{Depth-SIMS: Semi-Parametric Image and Depth Synthesis}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=iUGVt7BJ2l},
}
Figure